Too many requests - http 429

[gg4u]

Can't download - Too many requests - http 429

steps to reproduce

• I installed and upgraded edx-dl to last version
• I could not run any command, getting error 403. I fixed it with: https://github.com/coursera-dl/edx-dl/issues/468#issuecomment-395139398
• after the fix, I could run : edx-dl -u user@user.com -p **** --list-courses
• I tried to download a course, but got error 429 - too many requests.
• I tried to execute again edx-dl -u user@user.com --listcourses and see I now get also a 429 error.

I wonder if the credentials are not blocked by the backend.

See trace below:

edx_dl version 0.1.13
Building initial headers for future requests.
Getting initial CSRF token.
Traceback (most recent call last):
  File "/Users/XXX/Sites/miniconda3/bin/edx-dl", line 8, in <module>
    sys.exit(main())
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/site-packages/edx_dl/edx_dl.py", line 1006, in main
    headers = edx_get_headers()
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/site-packages/edx_dl/edx_dl.py", line 431, in edx_get_headers
    'X-CSRFToken': _get_initial_token(EDX_HOMEPAGE),
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/site-packages/edx_dl/edx_dl.py", line 168, in _get_initial_token
    opener.open(url)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Users/XXX/Sites/miniconda3/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

---

Edit

I also noticed this strange behaviour:

• first login passing the password : edx-dl -u user@user.com -p **** --list-courses will list the courses, OK
• If I try to downwload one of the courses by passing the password, it will give errror 403: edx-dl -u user@user.com -p **** COURSE_URL > error 403
• If I try to downwload one of the courses by NOT passing the password, terminal wil ask for password (I input the password of my user terminal, not the one of edx) and it will give errror 429: edx-dl -u user@user.com COURSE_URL > error 429
• After having raised error 429 one time, it will continue to raise it also for other commands, even the one that before worked fine: edx-dl -u user@user.com -p **** --list-courses > error 429

[gg4u]

Solution

Found a solution to the problem, which was a mismatched by a declared header at added line 166 ( see comment https://github.com/coursera-dl/edx-dl/issues/468#issuecomment-395139398 ) : added line opener.addheaders = [('User-agent', 'Mozilla/5.0')]

and the header used in edx_get_headers() :

make sure you use the same User-agent, otherwise the user agent in the first token will mismatched with future requests.