search

melisMirza / Covitter

Repository of SWE 573 project
0 stars 0 forks source link

snscrape Does Not Work on Heroku - find alternative #44

Closed melisMirza closed 3 years ago

melisMirza commented 3 years ago

Due to AWS policies, connection to snscrape servers are banned. The date search of the application does not work because of it. Find an alternative way for date search.

melisMirza commented 3 years ago

It is possible to access to Heroku postgres db from local machines, so it will be sent using a script that collects ID's from the snscrape and gets the content from Twitter API then send that to Heroku. The data will not be collected upon request but will be sent as bulk this way.

melisMirza commented 3 years ago

A scheduled work populated the DB with at least 2000 post everyday for the year.