search

bvdwiel / loadtest

Collection of scripts for load testing websites.
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

SQLite import is dog-slow #1

Closed bvdwiel closed 7 years ago

bvdwiel commented 7 years ago

Importing the results from the loadtest into SQLite is dog-slow at the moment. This is because every single row gets inserted by spawning a completely new sqlite3 command shell, opening the database, running the INSERT statement, closing the database and exiting the command shell again. Over.. and over.. and over. On my laptop this means waiting 2.5 minutes for 1500 records to insert. Barely tolerable in small test cases but useless for production unless allowed to run overnight.

SQLite itself isn't slow at doing a bazillion INSERTs as long as they're wrapped in a single transaction. Fix this issue by composing the whole import in some intermediate state (temp file?) and push that into SQLite in a single operation.

bvdwiel commented 7 years ago

Script writes the entire set of INSERT statements for a single results file to queryfile.sql now, encapsulates it in a transaction. The commit to DB is fairly instantaneous now, preparation of queryfile.sql still does take a lot of time but it's much more acceptable now. Run takes on average 25 minutes now.