Open damian0604 opened 3 years ago
The error when the scraper fails indicates that it scrolled too far/not enough (had that a lot when figuring out timestamp issues) or that it went too fast and the page wasn't loaded yet. Maybe you can run it again to check when exactly it fails so we can see what the reason was?
TODO: I will change the function to
yield
instead ofreturn
the elements,think of some error handling, and immediately dump tojsonlines
instead of one big json at the end so that we do not loose all the data we already got.