[BUG] Unable to load cookies [Changes on Udemy End]

[parasguglani1]

Describe the bug Unable to load cookies. Tried different VPNs
one is giving Exception in redeem courses: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version. and other is Exception in redeem courses: 'NoneType' object is not subscriptable To Reproduce

Steps to reproduce the behavior: execute udemy_controller --debug

Expected behavior Should load the data from file

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information): OS: Windows 10 Browser: Microsoft Edge Version 92.0.902.84 (Official build) (64-bit) Version: release V 3.1.0 VPN: Proton Additional context I created new account with VPN because of issue #312 and it was working and got enrolled in 148 courses but today I am getting these two errors.

[fakeid30]

@cullzie I can't reproduce this. can you?

[Pos7alDude]

I got the same issue ('NoneType' object is not subscriptable) on Manjaro Linux. Yesterday it worked perfect. Deleted and re-created settings.yaml and cloned the repository again but doesn't solve the issue. I dit not run any updates after last use of the tool. A friend thinks meybe udemy has changed something on their site, maybe even one div may cause this error. My settings.yaml: udemy: categories: [Development, Design, IT & Software, Software Engineering, Photography & Video, Business, Office Productivity, Personal Development] email: xxxxxxxxxxx@xxxxxxxxxx.xx languages: [German, English] password: xxxxxxxxxxxxxxxxx zipcode: '53773'

app.log: 2021-09-01 09:37:00,558 - udemy_enroller - INFO - settings : Loading existing settings 2021-09-01 09:37:00,571 - udemy_enroller - INFO - udemy : Loading cookies from file 2021-09-01 09:37:00,629 - udemy_enroller - ERROR - runner : Exception in redeem courses: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version. 2021-09-01 09:37:06,972 - udemy_enroller - INFO - settings : Loading existing settings 2021-09-01 09:37:06,985 - udemy_enroller - INFO - udemy : Loading cookies from file 2021-09-01 09:37:09,084 - udemy_enroller - INFO - udemy : Caching cookies for future use 2021-09-01 09:37:09,085 - udemy_enroller - INFO - udemy : Loading existing course details 2021-09-01 09:37:09,206 - udemy_enroller - INFO - udemy : Retrying login 2021-09-01 09:37:09,206 - udemy_enroller - INFO - udemy : Deleting cookies 2021-09-01 09:37:09,207 - udemy_enroller - INFO - udemy : Loading cookies from file 2021-09-01 09:37:10,036 - udemy_enroller - ERROR - runner : Exception in redeem courses: 'NoneType' object is not subscriptable

[ninjatall12]

i have gotten the same issue i think udemy did an update that requires people who use user-based logins to solve captcha i think the best way to solve this in the future is to use a cookie based auth. unfortunately im not familiar with web scraping so i can't help here.

[ninjatall12]

this is present with all udemy scrapers so its a udemy thing.

[cullzie]

@fakeid30 I am seeing something similar at the moment. Udemy have changed something on their side

[fakeid30]

ok now i can reproduce, also getting a lot of comments on social media.

[cullzie]

So there are 2 errors which show up here but they are both linked to the captcha appearing:

Cloudflare version 2 Captcha challenge and Exception in redeem courses: 'NoneType' object is not subscriptable

The fix isn't straightforward as we need a way to bypass the captcha which isn't currently a free service anywhere. Unless there is another way to get around the captcha the script may no longer be able to enroll in udemy courses :disappointed: If anyone has suggestions or other possible solutions please post them here.

[Pos7alDude]

The error Cloudflare version 2 Captcha challenge is not new to me. It was always solved by running the script again until the 'NoneType' error occured.

[c-hew]

@Pos7alDude running the script several times, results in either Cloudflare version 2 captcha challenge or exception in redeem... as described by cullzie, seems like it is not a solvable problem at the moment.

[cullzie]

So it looks like having it login in isn't the problem. The issue is when we try to access any of the endpoints with the generated cookie. I've tried with the curl request generated from the chrome dev tools and still hit the same issue.

[Pos7alDude]

@owlg Exactly what you write is currently the case. I wanted to clarify with my post that only the "captcha" message already occurred in the past from time to time. Now either the 'captcha' message or the 'NoneType' error comes up. So the errors don't necessarily have to be related. But cullzie seems to have figured that out already. So if the 'NoneType' error is cookie-related, it might be solvable. The captcha seems to be bypassable by re-running the script a few times.

[hoeni]

@Pos7alDude I hope you're right with the cookie-related error. But often big sites roll out new features (like a scraping protection) in several steps (blue/green deployment). So for a certain time only some servers have the change, e.g. to make performance or compatibility tests before it gets rolled out 100% in the end.

[ninjatall12]

This is what you get

[cullzie]

So no requests outside the browser seem to work at the moment. Previously we couldn't login on the browser hence we had to switch to REST requests.

We might need to look at a hybrid approach where we login via REST (which still seems to work) and then load those cookies into the browser. Not sure if this will work but I don't see another way at the moment. Ideally we'd go fully REST or fully browser based but the net keeps closing in on what is possible as they reduce accessibility

[cullzie]

Has the script started working for anyone else? Its working for me now with no modifications :confused: Possibly the changes have been rolled back or there was some issue with the release causing problems which has been fixed.

[basilioss]

Yes! It's also working for me now

[parasguglani1]

Has the script started working for anyone else? Its working for me now with no modifications 😕 Possibly the changes have been rolled back or there was some issue with the release causing problems which has been fixed.

working for me as well without VPN.

[fakeid30]

@cullzie not working for me, I get the captcha and another error (related to captcha, just a huge dump of broken ref) on a cyclic basis. Weird.

[cullzie]

@fakeid30 Looks like its regional deployments then. Not sure if they are rolling back or fixing issues at the moment though. We'll have to wait a bit and see if this gets to everyone

[fakeid30]

@cullzie false negative from my part. I was experimenting with the idea of hybrid login in local and today I ran the local branch instead of the develop branch from here.

The dev branch is working for me too.

But currently, if you input wrong password, it shows the captcha error. Just a small QoL.

[hoeni]

For me, the master branch wasn't working, but the dev branch worked (regular, German IP). It took a strange, long time for Loading existing course details, though, but went on.

[ninjatall12]

Afaik its returned bat to normal now but who knows if it will stay like this for long.

[parasguglani1]

it was working for me till yesterday but getting the same Cloudflare error today.

[ninjatall12]

it was working for me till yesterday but getting the same Cloudflare error today.

hmm the update might depend on the region I reside in the UK I'm assuming the Cloudflare error can only happen once a day so try to use it at the same time you used it yesterday you should be Cloudflare unblocked.

[Pos7alDude]

The Cloudfare captcha error appears very often here (Germany) but, as I wrote above, it doesn't matter becaus you can simply re-run the script (sometimes you need three or even four attempts) to get it working again. For me it's okay.

[cullzie]

Cloudflare captcha is a separate issue to the Udemy apis not working. Generally its because you've made too many requests within a certain time frame or you might be using a vpn that has a know vpn ip range to cloudflare.

For me the script is working at the moment

@Pos7alDude Would it make sense to have a bigger retry amount for udemy login? At the moment we re-try once. We could increase this which would save having to re-run the script multiple times. Not sure if it would work but if its something a lot of people are seeing it might be worth a try

[Pos7alDude]

@cullzie Maybe that would make sense for those who argue about the error message. But I'm fine as long as it works.

[Pos7alDude]

Now the "NoneType" error is occuring again since yesterday. No chance to run the script, tried multiple times.

[cullzie]

I think this is actually the 403 error. If you update to the latest release you should no longer have the NoneType error.

I have started seeing this again today and as far as I can see it isn't a straightforward fix. Even curl commands with all info from the browser is hitting issues

403 Client Error: Forbidden for url: https://www.udemy.com/api-2.0/https://www.udemy.com/api-2.0/users/me/subscribed-courses/?fields%5Bcourse%5D=title,completion_ratio&is_archived=false&ordering=-last_accessed&page=1&page_size=100

[Pos7alDude]

Now I tried several things. If I run udemy_enroller after pip install udemy-enroller I get either the cloudflare or the NoneType error. If I run python3 /path-to-enroller/udemy_enroller.py (cloned from git) I get the 403 error.

Okay sorry that was my fault. I hat first to run pip uninstall udemy-enroller and then pip install udemy-enroller to get the new version running this way. Just running pip install doesn't update the version

[cullzie]

@Pos7alDude You can run pip install <package_name> -U to upgrade a package

[cullzie]

Closing this as a duplicate of #327