prefilter in pandas, use one sqlalchemy session, and use futures to c…

mas-4 / maudlin2

A new sentiment analyzer

MIT License

2 stars 0 forks source link

Closed mas-4 closed 2 months ago

mas-4 commented 2 months ago

Prefilter headlines in pandas for updates
Use futures for thread management on scrapers so that we don't get stuck in a queue waiting for one thread to finish
Use a single sqlalchemy session instead of committing for every change

Previous commits took me from 30 minutes to 15. This one takes me from 15 to 3