Slowness on `stepNoCB`

[herberteuler]

I have a Haskell program that inserts data into an SQLite file. This program runs fairly slowly, it processes ~3900 rows in ~3 seconds.

I have also written a Python counterpart, and it processes the same dataset in ~0.16 seconds, or about 18 times faster.

After a lot of experiments, I simplified the core part of the program to the following function, which isolates the other factors by only using necessary functions provided by the library:

          add stmt time stock = do {-# SCC bind #-} bind stmt $ toFields time stock
                                   {-# SCC step #-} stepNoCB stmt
                                   {-# SCC reset #-} reset stmt

The profile shows that stepNoCB consumes most of the time:

         step                                    QuantRating                                  src/QuantRating.hs:74:53-65                                     12395        3903    0.0    0.0    89.0    0.1
          stepNoCB                               Database.SQLite3                             Database/SQLite3.hs:(417,1)-(418,83)                            12396        3903    0.0    0.0    89.0    0.0
           checkError                            Database.SQLite3                             Database/SQLite3.hs:277:1-54                                    12400        3903    0.0    0.0     0.0    0.0
           stepNoCB                              Database.SQLite3.Direct                      Database/SQLite3/Direct.hs:(461,1)-(462,47)                     12397        3903   89.0    0.0    89.0    0.0
            toStepResult                         Database.SQLite3.Direct                      Database/SQLite3/Direct.hs:(223,1)-(227,29)                     12398        3903    0.0    0.0     0.0    0.0
             decodeError                         Database.SQLite3.Bindings.Types              Database/SQLite3/Bindings/Types.hsc:(264,1)-(296,66)            12399        3903    0.0    0.0     0.0    0.0

I checked the definition of stepNoCB and it is merely a thin wrapper around an FFI call:

-- | <https://www.sqlite.org/c3ref/step.html>
--
-- Faster step for statements that don't callback to Haskell
-- functions (e.g. by using custom SQL functions).
stepNoCB :: Statement -> IO (Either Error StepResult)
stepNoCB (Statement stmt) =
    toStepResult <$> c_sqlite3_step_unsafe stmt

The system is macOS Mojave 10.14.6. Does anyone have any idea about why it is slow? Thanks.

[herberteuler]

It was because a transaction was missing.