search

RachitKamdar / Python-Scraper

0 stars 0 forks source link

data is not saved after the scraping is done #1

Open ltawseef opened 5 years ago

ltawseef commented 5 years ago

sir, I'm not able to save the data after the scraping process gets over. Some time the CPCB is not responding for any city so when the timeout occurs it stops and returns nothing. I have made some changes in the for loop which iterates through all cities in place of 'break' i made it as 'continue' so that if some of the city does not respond it skips and goes to the next city. Even after making the changes does not help me to save the data after it completes the scraping. Please help me out

RachitKamdar commented 5 years ago

Hi, yes the CPCB site does get down which hinders the code from working. Can't do much about it, your idea of using continue looks good. Can you provide more info on how you have been running the code (through command line/on an IDE), have you created a folder for download and have specified the path in the command line?

ltawseef commented 5 years ago

Sir, now its working fine, but i can only save 2 days data. if i will give startdate and enddate difference more than two days the data is not stored. Two days data comes in single page and when i give more then that the scrapper is going for second page and it displays the second page but will not save it. Can u plz check this out

santhoshsun55 commented 5 years ago

Hello, Thank you for sharing the script to download CPCB data. I have an issue with saving data. After scraping the data not saving n the folder. Kindly do needful.

Shruti0402 commented 3 years ago

Is it like the data is saved after it has crawled to all the states??