search

dmarx / twitterMonitor

Simple twitter event detection tool
1 stars 0 forks source link

migrate to a more robust database solution #36

Open dmarx opened 7 years ago

dmarx commented 7 years ago

Handling database locks from conflicting components is slowing down the tweet scraping mechanism, which is damaging the ability of the app to perform trend detection. Need to swap out sqlite3 for a database engine that handles locking better. To consider:

Additionally: the title daemone is writing titles directly into the entities table at the moment. this means that even if I resolve database level locks, the title_daemon and scraper will still encounter locking conflicts when trying to write into the entities table. Should factor out titles into a separate table to eliminate table level locking conflicts.

Should consider swapping out raw SQL for SQLAlchemy to make swapping out backends easier.

dmarx commented 7 years ago

As a temporary work-around, increased title fetching refresh time to two minutes. Local instance still has much better performance than AWS: need to figure out a way to scrape more tweets per second or something.