search

nerds-amp-mods / hackernewsclone

Hackernews - The Indian Version
1 stars 3 forks source link

Setup an aysnc io script once the scrapers are in place #9

Open sakethramanujam opened 4 years ago

sakethramanujam commented 4 years ago

Add an Asyncio scripts that runs all scrapers in intervals to periodically update the db with news articles.

Decide on a workflow mechanism to run async scripts. Whether it should be a client that the scraper should call or one single script that runs all the scrapers at once is up to you.

mm-git01 commented 4 years ago

a client that calls the scrappers and waits or sleeps for certain info would be a good workflow.Some asyc threads could wait on certain scrapers to update while some others who have already got the data get scheduled for the next run as soon as the data is written.

mm-git01 commented 4 years ago

@sakethramanujam have you found someone to write the scrapers yet ? this issue is being blocked by the scrapers.