dentarthur
commented
8 years ago (sigh) above log bizarrely used previous version in ../edx-dl.bak/edx_dl/
Now followed ##FIXME in README.MD (which should actually be fixed). Installed pandoc, created README.rst and was actually able to build and install with setup.py to create /usr/bin/edx-dl
(should just be via pypi).
Steps to reproduce
Tell us how to reproduce this issue. Please provide us the course URL, and the specific subsection or unit if possible.
edx-dl -i --sequential -u [xxx] -p [xxx] --cache -o '//home/admin/Documents/CURRENT/edx' -s https://courses.edx.org/courses/MITx/18.01.3x/1T2016/info
Actual Behaviour
Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
Downloading Calculus 1C: Coordinate systems and infinite series [MITx/18.01.3x/1T2016]
Downloading 4 section(s)
Section 1: Getting started
Overview and logistics
Tutorial on using the edX platform
Getting to know you
Entrance survey
Section 2: Unit 1: Coordinate systems
Parametric Curves
Polar Coordinates
Homework 1 Part A
Homework 1 Part B
Section 3: Unit 2: Infinite Series
L'Hospital's Rule
Improper integrals
Infinite series
Taylor series
Homework 2 Part A
Homework 2 Part B
Section 4: Exam Preparation and Survey
None
Exit Survey
loading 0 urls from cache [edx-dl.cache]
Extracting all units information in sequentially.
Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/getstarted/logistics-sequential/'
Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/getstarted/edxtutorial-sequential/'
Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/getstarted/diag-sequential/'
Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/getstarted/survey-sequential/'
Traceback (most recent call last):
File "/usr/bin/edx-dl", line 9, in
Looks like same problem to me.
But previously it did (in first trace above):
"loading 266 urls from cache [edx-dl.cache]"
Now with update (second trace):
"loading 0 urls from cache [edx-dl.cache]"
Changed from //home... to "--cache -o '/home/..."
Still only loaded 0 urls from cache.
But now it looks like it is painfully working through previously downloaded files as before failure.
Will take a while. I'm posting this now to assume problem gone with update so this issue is only about the need to fix pypi or README unless I provide another trace after sleep.
iemejia
balta2ar
🚨Please review the Troubleshooting section before reporting any issue. Don't forget also to check the current issues to avoid duplicates.
Cannot see link for "Troubleshooting", Reviewed current for duplicates saw none.
Previously working now reliably fails for more than a week
Describe your issue here. Repeated connection timeout
Your environment
README.MD said
pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4>=4.1.3 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): html5lib>=1.0b2 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): six>=1.5.0 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): youtube-dl>=2015.05.20 in /usr/lib/python2.7/site-packages (from -r requirements.txt (line 4)) Cleaning up...
So updated with:
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade edx-dl
Collecting edx-dl Could not find a version that satisfies the requirement edx-dl (from versions: ) No matching distribution found for edx-dl
Ditto no pypi for python 2.7 so need to update pypi or revise README.
Tried: $ python setup.py Traceback (most recent call last): File "setup.py", line 32, in
long_description=open('README.rst', 'r').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'
Gave up and just extracted edx-dl without setup.
Steps to reproduce
Tell us how to reproduce this issue. Please provide us the course URL, and the specific subsection or unit if possible.
python edx-dl.py -i --sequential -u [xxx] -p [xxx] --cache -o '//home/admin/Documents/CURRENT/edx' -s https://courses.edx.org/courses/MITx/18.01.3x/1T2016/info
Expected behaviour
Tell us what should happen. Later units should download after painfully slowly checking files already downloaded previously for units 0, 1 and 2 of Calculus 1C course at edx.
Repeatedly would not download with same error for more than a week. Does not get to point of painfully checking previous downloads.
Same after updating to latest commit as above.
Actual behaviour
Tell us what happens instead. If the script fails, please copy the entire output of the command or the stacktrace (don't forget to obfuscate your username and password). If you cannot copy the exception, attach a screenshot.
Building initial headers for future requests. Getting initial CSRF token. Found CSRF token. Logging into Open edX site: https://courses.edx.org/login_ajax Extracting course information from dashboard. Downloading Calculus 1C: Coordinate systems and infinite series [MITx/18.01.3x/1T2016] Downloading 4 section(s) Section 1: Getting started Overview and logistics Tutorial on using the edX platform Getting to know you Entrance survey Section 2: Unit 1: Coordinate systems Parametric Curves Polar Coordinates Homework 1 Part A Homework 1 Part B Section 3: Unit 2: Infinite Series L'Hospital's Rule Improper integrals Infinite series None Homework 2 Part A Homework 2 Part B Section 4: Exam Preparation and Survey Final exam preparation Exit Survey loading 266 urls from cache [edx-dl.cache] Extracting all units information in sequentially. Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/unit10/ps10A-sequential/' Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/unit10/ps10B-sequential/' Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/prep_survey/prep-sequential/' Processing 'https://courses.edx.org/courses/MITx/18.01.3x/1T2016/courseware/prep_survey/exit-sequential/' Traceback (most recent call last): File "edx-dl.py", line 6, in
edx_dl.main()
File "/home/admin/Documents/study-other/downloader/edx-dl.bak/edx_dl/edx_dl.py", line 968, in main
all_units = extract_all_units_with_cache(all_urls, headers, extractor=extractor)
File "/home/admin/Documents/study-other/downloader/edx-dl.bak/edx_dl/edx_dl.py", line 863, in extract_all_units_with_cache
new_units = extractor(new_urls, headers)
File "/home/admin/Documents/study-other/downloader/edx-dl.bak/edx_dl/edx_dl.py", line 418, in extract_all_units_in_sequence
units = [extract_units(url, headers) for url in urls]
File "/home/admin/Documents/study-other/downloader/edx-dl.bak/edx_dl/edx_dl.py", line 403, in extract_units
page = get_page_contents(url, headers)
File "/home/admin/Documents/study-other/downloader/edx-dl.bak/edx_dl/utils.py", line 58, in get_page_contents
result = urlopen(Request(url, None, headers))
File "/usr/lib64/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1224, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1186, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>