Closed CathyChang1996 closed 5 years ago
@CathyChang1996
After testing, I think the main reasons here are scraping too fast
and the scroll times
may not by 1 time each page.
Solutions :
for loop
, like 2 seconds.Thanks a lot!! Hope this time would work!!
Troubleshooting
Describe your environment
Describe your question
I found that when I scrape the pages in mafengwo, the code cannot completely go through all the pages. Here is the page link:http://www.mafengwo.cn/wenda/u/5017124/answer.html
The total number of the pages is 85, but the Jupiter notebook completed its running when it's on Page 44, and all the data I scraped copied, so the number of all data seems right but the real content duplicates.
I wonder if it is a kind of anti-crawling process.
Describe the efforts you have spent on this issue