search

ISNIT0 / society-scraper

MIT License
0 stars 0 forks source link

Bedforshire loops #61

Closed Kickingmen closed 4 years ago

Kickingmen commented 4 years ago

Bedfordshire loops the first 8 socs with every load more option, description is in the comments.

Kickingmen commented 4 years ago

Will scrape the first 8 socs, then "load more" then will scrape the first 8 again and then the next 8.

so if there are 9 "load more" options, the first 8 socs get scraped 72 times, the 2nd 8 socs, 64 times etc.

This is an issue for over 40 universities. Dramatically increasing the time to scrape.

Kickingmen commented 4 years ago

Fixed with :last-of-type