Context
The COPA data being scraped contains a significant number of rows, and existing rows within our database must be checked for duplicates every time a scrape occurs. As the number of rows in the database grows, the efficiency of the loading strategy will be significant.
Problem
The current design of the loader has not taken into account the efficiency of the loader.
Solution
Use benchmark tests to compare solutions. See commits 3fc1106 and fba8524.
Increase efficiency of database loading
Context The
COPAdata being scraped contains a significant number of rows, and existing rows within our database must be checked for duplicates every time a scrape occurs. As the number of rows in the database grows, the efficiency of the loading strategy will be significant.Problem The current design of the loader has not taken into account the efficiency of the loader.
Solution Use benchmark tests to compare solutions. See commits 3fc1106 and fba8524.