Nubilefilms must have some kind of ddos system (I'm just guessing) because I think I scrapped too much from them and got my IP banned. Changed my IP and working again. Just scrapping that site much slower so I dont get banned again. Is there a way to do the automated scrape but slower?
I do not know what the frequency should be for Nublies. I have just been doing one every couple minutes for a hour a day. It's probably not an issue for someone that gets a new video every day or two but if you're trying to scrape a large collection it could be an issue. I just tried Nublies from the "banned" IP an it worked today. So I guess they aren't doing a permanent ban but I wouldn't be surprised if they did if you keep scraping the site for a large collection. Would it be possible to create an adjustable timer or something that could be set by the user? Also, you can add NFBusty to your NubliesFilms scraper. I have found that it scrapes that site too.
Original request: #87
Nubilefilms must have some kind of ddos system (I'm just guessing) because I think I scrapped too much from them and got my IP banned. Changed my IP and working again. Just scrapping that site much slower so I dont get banned again. Is there a way to do the automated scrape but slower?
I do not know what the frequency should be for Nublies. I have just been doing one every couple minutes for a hour a day. It's probably not an issue for someone that gets a new video every day or two but if you're trying to scrape a large collection it could be an issue. I just tried Nublies from the "banned" IP an it worked today. So I guess they aren't doing a permanent ban but I wouldn't be surprised if they did if you keep scraping the site for a large collection. Would it be possible to create an adjustable timer or something that could be set by the user? Also, you can add NFBusty to your NubliesFilms scraper. I have found that it scrapes that site too.