search

hammady / wwpray

A website that shows the prayer times for a preconfigured list of masjids in a tabular format. Visitors can subscribe for email notifications.
https://wwpray.net
MIT License
1 stars 0 forks source link

Optimize scraper to run scraping in parallel #39

Closed hammady closed 1 year ago

hammady commented 1 year ago

Currently, the scraper scrapes websites sequentially. There is a lot of wasted time in the function waiting for network (probably 99%!). This increases the cost of the function even though it is limited to 20 websites only. It would be better to do async requests and write the final file once all scrapers finish.

hammady commented 1 year ago

Before:

image

After:

image

Note that in the last screenshot, a failure with retry is simulated for one of the sources.