search

lmolteno / nsn

NCEA Standards Navigator
https://www.notion.so/Fast-access-NZQA-d4f21847f9174bc2954bd6a3e8205363
1 stars 1 forks source link

Performance issues with the scraping and combination scripts #34

Closed lmolteno closed 3 years ago

lmolteno commented 3 years ago

These are exaggerated by the addition of resources to my database. I think that this would be a good place to implement a Rust solution. Multithreading!

lmolteno commented 3 years ago

It only takes a minute to do the full scrape, and with optimisation I did in #37, it's pretty insignificant once the server is running, the only down-time in the database is during the actual entry through psycopg commits.