[BUG] Time out / Forbidden

[xoco70]

When using develop branch, everything works smooth, til page 4, after 12 links retrieved. Then, I get timeouts, and page is shown as forbidden in the browser/

Maybe inserting a timeout should do the trick

Additional context Ubuntu 20.04 Chrome

[fakeid30]

@xoco70 did you go to page 5 and check if there are any valid links there ?

What is supposed to happen is , the script has ran out of things to enroll and constantly hitting links that you are subscribed to. You can increase this threshold with a command line parameter.

Let me know if this solves your issue. I am closing this right now but feel free to comment back if you have additional finding.

[xoco70]

How should I go to page 5 manually ?

I think forbidden is coming before the end of the loop.

After that, when logging to a private window with another account, I must go through ReCaptcha again.

Here is the log of my session. At the end, I closed my browser, as all url were giving "Forbidden"

Got 45 links from https://comidoc.net in 0.39 seconds
Page: 1 of 650 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 2.05 seconds
Successfully enrolled in: Docker Masterclass for Machine Learning and Data Science | Udemy
Successfully enrolled in: The Complete HTML , CSS , Bootstrap :Build Hands On Projects | Udemy
Successfully enrolled in: The Complete 3D Artist: Learn 3D Art by Creating 3 Scenes | Udemy
Successfully enrolled in: Machine Learning for Kids and Beginners | Udemy
Successfully enrolled in: Create a Members Only Blog using PHP, MySQL, & AJAX | Udemy
Successfully enrolled in: Setup a Virtual Web Server using Linode or Digital Ocean | Udemy
Successfully enrolled in: The Complete HTML5 Course: From Beginning to Expert | Udemy
Successfully enrolled in: Wordpress For Beginners | Udemy
Successfully enrolled in: Python Demonstrations For Practice Course | Udemy
Successfully enrolled in: Learning Python 3 Network Programming for Network Engineers | Udemy
Successfully enrolled in: Introduction to Domain Names and Web Hosting - Quick Guide | Udemy
Successfully enrolled in: Scrum Fundamentals for Scrum Master and Agile Projects- 2020 | Udemy
Successfully enrolled in: Le guide complet des 20 meilleurs plugins WordPress | Udemy
tutorialbar scraper is running
Page: 2 of 650 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 2.29 seconds
tutorialbar scraper is running
Page: 3 of 650 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.59 seconds
tutorialbar scraper is running
Page: 4 of 650 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.65 seconds
Successfully enrolled in: Create Your Own Augmented Reality Application | Udemy
Timeout on link: https://www.udemy.com/course/python-introduction-to-data-science-and-machine-learning-a-z/?couponCode=8FB266BFA6D6C32BADCB
Timeout on link: https://www.udemy.com/course/how-to-make-passive-income-without-experience-or-a-website/?couponCode=STAYHOME1220201
tutorialbar scraper is running
Page: 5 of 650 scraped from tutorialbar.com
Stopping loop. We have reached max number of pages to scrape: 5
tutorialbar scraper complete
Got 12 links from https://www.tutorialbar.com in 1.19 seconds
Timeout on link: https://www.udemy.com/course/pmp-certification-exam-2020-300-practice-questions-topic-1/?couponCode=6DD743A901B123C02A42
Webdriver exception on link: https://www.udemy.com/course/sd-card-interfacing-with-arduino/?couponCode=STAYHOME1220201
Webdriver exception on link: https://www.udemy.com/course/comptia-a-plus-certificate-motherboards-processors-memory/?couponCode=EBCF3E6DC2CD5980ACE4
Webdriver exception on link: https://www.udemy.com/course/turn-videos-into-books-blog-posts-and-transcripts-amazon-kdp-book-sell/?couponCode=STAYHOME1220201
Webdriver exception on link: https://www.udemy.com/course/setsa-selfempowerment/?couponCode=COMMUNITY
Webdriver exception on link: https://www.udemy.com/course/the-complete-journaling-course-build-self-awareness-habits/?couponCode=3300B302CADFD2464263
Webdriver exception on link: https://www.udemy.com/course/protect-me-device-pendant-warn-you-when-you-touch-your-face-covid19/?couponCode=BAA031B28330508CB8B4
Webdriver exception on link: https://www.udemy.com/course/troubleshoot-electronics-projects-devices-check-maintenance-electronic/?couponCode=STAYHOME1220201
Webdriver exception on link: https://www.udemy.com/course/certified-lean-six-sigma-black-belt-training-e/?couponCode=BLACK20
Webdriver exception on link: https://www.udemy.com/course/hare-krishna-complete-resume-writing-job-search-masterclass/?couponCode=MERRYCHRISTMAS100
Webdriver exception on link: https://www.udemy.com/course/numbers-in-turkish-1-100/?couponCode=A6BA62F1A318B2586755
Webdriver exception on link: https://bit.ly/3mBUnGz
All scrapers complete
Closing browser

[xoco70]

In this log, I already enrolled the first timeout link

Timeout on link: https://www.udemy.com/course/python-introduction-to-data-science-and-machine-learning-a-z/?couponCode=8FB266BFA6D6C32BADCB

But the second timeout has not been enrolled:

Timeout on link: https://www.udemy.com/course/how-to-make-passive-income-without-experience-or-a-website/?couponCode=STAYHOME1220201

[fakeid30]

@cullzie can you take a look at this?

[cullzie]

Hey @xoco70, The only time I've seen timeouts like this are when my internet connection has been poor or we are trying to navigate to one of the bit.ly lnks. As for the webdriverexceptions after that I would say the browser window had been closed. Can you give some info on what is showing as forbidden. Looking at the logs isn't giving much

[xoco70]

Yep, for the webdriverexception, I closed the browser, because it was giving a timeout.

For the timeout, I have not seen them since I use develop branch, so... Maybe it has been solved already... I'll let you know if I see it again !

[xoco70]

I've just seen this issue again in develop branch, so I reopen the issue.

I could enroll manually for each timedout link.

Maybe the forbidden is due to too many pages browsed in a small amount of time. When script spend some time enrolling 1 course, it doesn't happen, but when it scrap a lot of script quickly with no success is when I think it is happening

➜  Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE git:(develop) ✗              python3 udemy_enroller.py --browser=chrome                
Loading existing settings

[WDM] - Current google-chrome version is 87.0.4280
[WDM] - Get LATEST driver version for 87.0.4280
[WDM] - Driver [/home/julien/.wdm/drivers/chromedriver/linux64/87.0.4280.88/chromedriver] found in cache
tutorialbar scraper is running
comidoc scraper is running
Page: 1 of 657 scraped from tutorialbar.com
comidoc scraper complete
Got 45 links from https://comidoc.net in 10.16 seconds
Got 12 links from https://www.tutorialbar.com in 15.72 seconds
Successfully enrolled in: Artificial Neural Networks for Business Managers in R Studio | Udemy
Successfully enrolled in: Create your own GUI Music Player using Pygame & Tkinter | Udemy
Successfully enrolled in: Introduction to Database Engineering | Udemy
Successfully enrolled in: Logistic Regression in R Studio | Udemy
Successfully enrolled in: 1 Hour CSS | Udemy
Timeout on link: https://www.udemy.com/course/digital-character-design-creations/?couponCode=F6291D96418988D08106
Timeout on link: https://www.udemy.com/course/create-drop-servicing-website-with-wordpress/?couponCode=DEC2020PROMO
Timeout on link: https://www.udemy.com/course/youtube_success/?couponCode=FORMYSUBS
Timeout on link: https://www.udemy.com/course/optimization-in-python/?couponCode=C7DB28483CEB2B040BF9
Timeout on link: https://www.udemy.com/course/ultimate-vba-macros-training-course/?couponCode=FREEVBA
Timeout on link: https://www.udemy.com/course/10-x-schneller-lernen-lerntechnik-training-lernen-lernen/?couponCode=6C845E56F0769238F612
tutorialbar scraper is running
Page: 2 of 657 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 12.68 seconds
Successfully enrolled in: Digital Character Design Creations | Udemy
tutorialbar scraper is running
Page: 3 of 657 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 11.50 seconds
Successfully enrolled in: 2020 Complete Pyomo Bootcamp: Python Optimization Beginners | Udemy
Timeout on link: https://bit.ly/34LGvn8
tutorialbar scraper is running
Page: 4 of 657 scraped from tuto
```rialbar.com
Got 12 links from https://www.tutorialbar.com in 11.10 seconds
tutorialbar scraper is running
Page: 5 of 657 scraped from tutorialbar.com
Stopping loop. We have reached max number of pages to scrape: 5
tutorialbar scraper complete
Got 12 links from https://www.tutorialbar.com in 11.44 seconds
All scrapers complete
Closing browser

[cullzie]

@xoco70 I'm finding it difficult to reproduce this issue. Timeouts are generally to do with a slow internet connection. Since its taking 10 secs for comidoc and 15 for tutorialbar it would suggest the internet speed is not great. You could increase the timeouts in udemy.py. So in every webdriverwait increase the timeout from 5/10 to 30 and see if that makes any difference. If speed is the issue we can create a config to control these settings so you don't need to edit the code

[xoco70]

Yep ! Not a big deal, I am only reporting it !

[BazioDEV]

@xoco70 I'm finding it difficult to reproduce this issue. Timeouts are generally to do with a slow internet connection. Since its taking 10 secs for comidoc and 15 for tutorialbar it would suggest the internet speed is not great. You could increase the timeouts in udemy.py. So in every webdriverwait increase the timeout from 5/10 to 30 and see if that makes any difference. If speed is the issue we can create a config to control these settings so you don't need to edit the code

by the way i also get the time out and forbidden while my internet speed was 1gbps/ 500mb

[cullzie]

@xoco70 @BazioDEV I have started seeing the Timeout issue a bit more. It looks like the link is loading as Add to cart for a split second before it changes to Already enrolled button. Then we are clicking the Add to Cart button even though we are already enrolled which is causing the Timeout to occur. I haven't seen any Forbidden exceptions

[fakeid30]

@xoco70 @BazioDEV This won't happen in our alpha branch anymore. closing this.