Small improvements to import performance

kfsone commented 2 months ago

There's a bunch of linting/fixing first, which helped me spot some actual problems. After that there's some tidy-up of eddbplugin that made it easier for me to reason about, and enabled me to find a balance between building big lists in memory to attack the db with vs making better use of the dbs own inevitable cache anyway.

this reduces import time from 40 minutes to 23 minutes on my windows pc.

please run this branch on other hardware yourself before merging.

eyeonus commented 2 months ago

Just got home from work, running a comparison check of trade import -P eddblink -O clean on this versus TD v11.0.3

eyeonus commented 2 months ago

NOTE: Downloading file 'listings.csv'.
NOTE: Requesting https://elite.tromador.com/files/listings.csv
NOTE: Downloaded   2.5GB of gziped data  22.6MB/s
NOTE: Processing market data from listings.csv: Start time = 2024-04-26 03:50:03.790430. Live = True

Live should not be true on this one.

eyeonus commented 2 months ago

36 minutes to completion with this branch, 11.0.3 is still at 88% on the main listings file

eyeonus commented 2 months ago

This branch:

NOTE: Processing market data from listings.csv: Start time = 2024-04-26 03:50:03.790430. Live = True
[================================================= ] 98% 43355318 / 44240120 NOTE: Optimizing database...
NOTE: Finished processing market data. End time = 2024-04-26 04:24:45.873538  
NOTE: Checking for update to 'listings-live.csv'.
NOTE: Downloading file 'listings-live.csv'.
NOTE: Requesting https://elite.tromador.com/files/listings-live.csv
NOTE: Downloaded  10.2MB of gziped data   7.4MB/s
NOTE: Processing market data from listings-live.csv: Start time = 2024-04-26 04:24:48.857846. Live = True
[================================================= ] 98% 184164 / 187922 NOTE: Optimizing database...
NOTE: Finished processing market data. End time = 2024-04-26 04:26:32.942536
NOTE: Import completed.

listings: ~35 minutes listings-live: ~2 minutes

Total: ~37 minutes TD v11.0.3:

NOTE: Processing market data from listings.csv: Start time = 2024-04-26 03:48:12.684702
NOTE: Finished processing market data. End time = 2024-04-26 04:34:42.974157 
NOTE: Checking for update to 'listings-live.csv'.
NOTE: Downloading file 'listings-live.csv'.
NOTE: Requesting https://elite.tromador.com/files/listings-live.csv
NOTE: Downloaded  10.3MB of gziped data   7.3MB/s
NOTE: Processing market data from listings-live.csv: Start time = 2024-04-26 04:34:45.877749
NOTE: Finished processing market data. End time = 2024-04-26 04:37:06.013385
NOTE: Import completed.

listings: ~46 minutes listings-live: ~3 minutes

Total: ~49 minutes

Total time savings: 12 minutes

eyeonus commented 2 months ago

I'm going to push a couple changes, check I didn't break anything, and then do the merge.

kfsone commented 2 months ago

Nice!

| remove unused methods

mmmmm - I love it when you talk dirty

eyeonus / Trade-Dangerous

Small improvements to import performance #131