This repo is a part of blog series on several web scraping projects where we will explore scraping techniques to crawl data from simple websites to websites using advanced protection.
Is it possible to add to the crawler the ability to crawl specific periods? right now, its running perfectly and crawl mostly 2020. Is there a way to add time based filter (for example for 2019)?
I wasnt sure where to add it in the code. If you can guid me where, it would be great!
Hi Amit, great cralwer!!
well done :)
Is it possible to add to the crawler the ability to crawl specific periods? right now, its running perfectly and crawl mostly 2020. Is there a way to add time based filter (for example for 2019)? I wasnt sure where to add it in the code. If you can guid me where, it would be great!
Thanks, Neta