search

ctjacobs / pyqso

PyQSO is a contact logging tool for amateur radio operators.
http://christianjacobs.uk/pyqso
GNU General Public License v3.0
79 stars 19 forks source link

Import ADIF records into database in one go #29

Closed ctjacobs closed 8 years ago

ctjacobs commented 9 years ago

Currently the slowest part of importing ADIF files seems to be the insertion of each record one-by-one into the sqlite database, with individual database commits. It would probably be faster to queue them up somehow and insert them all in one go.

ctjacobs commented 8 years ago

After gathering all the records to insert into one list, using executemany seems to bring a significant performance benefit when inserting all the records into the database. For an ADIF file of ~6,500 entries, the import time is reduced from ~150 seconds (with a for loop calling execute in each iteration) to ~7 seconds with one call to executemany on my laptop. Populating the actual log table (i.e. the GTK ListStore) seems to still take a while though.

Thanks to Timo OH7JHA for providing a relatively large ADIF file to test with.

ctjacobs commented 8 years ago

Sorted out an add_record bug introduced by 37685c9a4c214d29b11fd694f06909fd1a5495ae (see 794d9995870a9ba42b6f3c2b8ad37d81580d7fd1). I think this is fine now, so closing.