search

coursera-dl / edx-dl

A simple tool to download video lectures from edx.org (and other openedx sites)
GNU Lesser General Public License v3.0
1.93k stars 638 forks source link

edx AP Macroeconomics download starts ok, wanders off collecting youtubes #285

Open dentarthur opened 9 years ago

dentarthur commented 9 years ago

Appears to be confusing youtube recommendations of other videos that are displayed at sides of youtube pages with intended course videos. Presumably some recent youtube change so critical.

Command and partial listing of 403 items, totalling 11.1 GB downloaded so far instead of just s.03 of the course below. Version was installed 2015-07-30 and has worked ok for other edx courses since then except as noted in #265. Still going - alphabetical listing is presumably related to use of ls though I would not have expected it.

---getapmacro--- cd /home/admin/Documents/study-other/downloader/edx-dl/edx-dl

python edx-dl.py -i --sequential -u [...] -p [...] --quiet --cache -o '/run/media/admin/My Book/CURRENTedx' -s https://courses.edx.org/courses/course-v1:DavidsonNext+Mac_APccx+2T2015/info

$ cd '/run/media/admin/My Book/CURRENTedx' $ ls -l AP/ AP_Macroeconomics-_Challenging_Concepts/01-Getting_Started: total 63919 -rw-------. 1 admin admin 493 Aug 21 18:06 01-DVDDXN02T314-H010102_100-9Wg9qNWW1dY.en.srt -rw-------. 1 admin admin 4299084 Aug 16 2014 01-DVDDXN02T314-H010102_100-9Wg9qNWW1dY.mp4 -rw-------. 1 admin admin 3554 Aug 21 18:07 02-edX-FA12-cware-1_100-OEoXaMPEzfM.en.srt -rw-------. 1 admin admin 20540562 Apr 27 2014 02-edX-FA12-cware-1_100-OEoXaMPEzfM.mp4 -rw-------. 1 admin admin 2275 Aug 21 18:08 03-edX-FA12-cware-2_100-7m8pab1MfYY.en.srt -rw-------. 1 admin admin 13599881 Mar 9 2014 03-edX-FA12-cware-2_100-7m8pab1MfYY.mp4 -rw-------. 1 admin admin 2231 Aug 21 18:08 04-edX-FA12-cware-3_100-pflzzK47wEo.en.srt -rw-------. 1 admin admin 13055930 Feb 10 2014 04-edX-FA12-cware-3_100-pflzzK47wEo.mp4 -rw-------. 1 admin admin 2043 Aug 21 18:09 05-edX-FA12-cware-4_100-Q-rY8DIwYgg.en.srt -rw-------. 1 admin admin 13938614 Feb 8 2014 05-edX-FA12-cware-4_100-Q-rY8DIwYgg.mp4

AP_Macroeconomics-_Challenging_Concepts/02-Real_and_Nominal_Values: total 58617 -rw-------. 1 admin admin 378 Aug 21 18:09 01-DVDDXN02T314-I090104-5JEphUkAhtU.en.srt -rw-------. 1 admin admin 3329273 Jun 9 23:41 01-DVDDXN02T314-I090104-5JEphUkAhtU.mp4 -rw-------. 1 admin admin 4494 Aug 21 18:14 02-DVDDXN02T314-G090103-q3vk0-NP2is.en.srt -rw-------. 1 admin admin 29649800 Jul 23 19:07 02-DVDDXN02T314-G090103-q3vk0-NP2is.mp4 -rw-------. 1 admin admin 4823 Aug 21 18:18 03-DVDDXN02T314-G090205-wI30ehzc7bs.en.srt -rw-------. 1 admin admin 27022229 Jun 16 12:20 03-DVDDXN02T314-G090205-wI30ehzc7bs.mp4

AP_Macroeconomics-_Challenging_Concepts/03-Classical_and_Keynesian_Models: total 10279511 -rw-------. 1 admin admin 507 Aug 21 18:19 01-DVDDXN02T314-I060103-agbAo6f7tOM.en.srt -rw-------. 1 admin admin 3621417 Jun 10 03:58 01-DVDDXN02T314-I060103-agbAo6f7tOM.mp4 -rw-------. 1 admin admin 8692 Aug 21 18:27 02-DVDDXN02T314-G060106-QbAv6rBy5mA.en.srt -rw-------. 1 admin admin 50085900 Jun 9 13:35 02-DVDDXN02T314-G060106-QbAv6rBy5mA.mp4 -rw-------. 1 admin admin 9570 Aug 21 18:30 03-DVDDXN02T314-G060207-iNZ-Viu7U-I.en.srt -rw-------. 1 admin admin 49958293 Jun 16 12:27 03-DVDDXN02T314-G060207-iNZ-Viu7U-I.mp4 -rw-------. 1 admin admin 59964883 May 21 20:22 04-1940 - Axis gains momentum in World War II-Toss6Hzf5n0.mp4 -rw-------. 1 admin admin 107306351 Jun 18 15:15 04-1941 Axis momentum accelerates in WW2-X3bqQI7-sCg.mp4 -rw-------. 1 admin admin 47910236 May 16 21:22 04-1942 Tide turning in World War II in Europe-FqGLqdk9Sr0.mp4 -rw-------. 1 admin admin 46608745 May 20 05:45 04-1943 Axis losing in Europe-x9g82IA-IEU.mp4 -rw-------. 1 admin admin 61966247 May 20 15:10 04-1944 - Allies advance further in Europe-0UgQ0B_0hAg.mp4 -rw-------. 1 admin admin 65059300 Jun 4 21:43 04-1945 - End of World War II-U6RFAUmYU5o.mp4 -rw-------. 1 admin admin 26611865 Apr 10 16:49 04-2015 Econ Music Video Competition Results-AcCrIqEmV68.mp4 -rw-------. 1 admin admin 25915076 Jan 23 2015 04-2015 Economics Music Video Competition-u8zDEU_VYk4.mp4 -rw-------. 1 admin admin 6094 Aug 21 23:37 04-Adding mixed numbers-pxX07gUbIQQ.en.srt -rw-------. 1 admin admin 9011011 May 16 21:13 04-Adding mixed numbers-pxX07gUbIQQ.mp4 -rw-------. 1 admin admin 62753544 May 18 17:31 04-American progress in the Pacific in 1944--6XBnjFrWBg.mp4 -rw-------. 1 admin admin 17753 Aug 21 21:35 04-Antiparallel structure of DNA strands-0CQ5ls3Uc2Q.en.srt -rw-------. 1 admin admin 61628785 Jul 22 18:33 04-Antiparallel structure of DNA strands-0CQ5ls3Uc2Q.mp4 -rw-------. 1 admin admin 10864560 Apr 30 17:44 04-AP Economics Exam Review-kT0ddCwP1PU.mp4 -rw-------. 1 admin admin 19306762 Nov 21 2014 04-AP® Economics Workshops With Jacob Clifford-SSOYgkx7RA.mp4 -rw-------. 1 admin admin 6974 Aug 21 22:41 04-Approximating with powers of 10 Numbers and operations 8th grade Khan Academy-Xl724v0CU7E.en.srt -rw-------. 1 admin admin 28194662 Apr 30 04:41 04-Approximating with powers of 10 Numbers and operations 8th grade _ Khan Academy-Xl724v0CU7E.mp4 -rw-------. 1 admin admin 12589 Aug 21 22:15 04-Arrhenius definition of acids and bases-Y4HzGldIAss.en.srt -rw-------. 1 admin admin 34740400 Aug 20 04:35 04-Arrhenius definition of acids and bases-Y4HzGldIAss.mp4 -rw-------. 1 admin admin 7707 Aug 21 19:52 04-ATP hydrolysis mechanism--KE7jTXwNYs.en.srt -rw-------. 1 admin admin 18447180 Aug 17 19:32 04-ATP hydrolysis mechanism--KE7jTXwNYs.mp4 -rw-------. 1 admin admin 18676 Aug 21 22:17 04-Autoionization of water-NUyYlRxMtcs.en.srt -rw-------. 1 admin admin 34265374 Jul 8 18:15 04-Autoionization of water-NUyYlRxMtcs.mp4 -rw-------. 1 admin admin 788 Aug 21 23:31 04-Beginning of World War II--kKCjwNvNkQ.cs.srt -rw-------. 1 admin admin 70341867 May 22 16:15 04-Beginning of World War II--kKCjwNvNkQ.mp4 -rw-------. 1 admin admin 15623 Aug 21 22:16 04-Bronsted-Lowry definition of acids and bases-XVOU9c3crbc.en.srt -rw-------. 1 admin admin 28895410 Aug 5 23:09 04-Bronsted-Lowry definition of acids and bases-XVOU9c3crbc.mp4 -rw-------. 1 admin admin 10909 Aug 21 22:14 04-Buffer system-gjKmQ501sAg.en.srt -rw-------. 1 admin admin 20164306 Aug 17 15:56 04-Buffer system-gjKmQ501sAg.mp4

[...]

-rw-------. 1 admin admin 13774 Aug 21 20:13 04-Uniporters, symporters and antiporters--aGYytZ7K7M.en.srt -rw-------. 1 admin admin 48538845 Aug 5 20:40 04-Uniporters, symporters and antiporters--aGYytZ7K7M.mp4 -rw-------. 1 admin admin 7507 Aug 21 22:41 04-Using proportion to solve for variable-s8_14yxp1lQ.en.srt -rw-------. 1 admin admin 18815091 Jul 23 02:27 04-Using proportion to solve for variable-s814yxp1lQ.mp4 -rw-------. 1 admin admin 3736 Aug 21 23:02 04-Using reflection tool Transformations Geometry Khan Academy-RLyXTj2jc4.en.srt -rw-------. 1 admin admin 6877293 Jun 18 16:14 04-Using reflection tool Transformations Geometry Khan Academy-RLyXTj2jc4.mp4 -rw-------. 1 admin admin 2349 Aug 21 23:02 04-Using rotation widget Transformations Geometry Khan Academy-UKM8BnpZr94.de.srt -rw-------. 1 admin admin 2243 Aug 21 23:02 04-Using rotation widget Transformations Geometry Khan Academy-UKM8BnpZr94.en.srt -rw-------. 1 admin admin 6067237 May 18 12:03 04-Using rotation widget Transformations Geometry Khan Academy-UKM8BnpZr94.mp4 -rw-------. 1 admin admin 16160 Aug 21 22:23 04-Water as a solvent-lCvBp73ZJ-A.en.srt -rw-------. 1 admin admin 38789134 Jul 23 18:07 04-Water as a solvent-lCvBp73ZJ-A.mp4 -rw-------. 1 admin admin 22819295 May 29 17:28 04-What is the domain of a function Functions Algebra I Khan Academy--DTMakGDZAw.mp4 -rw-------. 1 admin admin 11132 Aug 21 22:34 04-What is the range of a function Functions Algebra I Khan Academy-96uHMcHWD2E.en.srt -rw-------. 1 admin admin 20758602 Jun 27 20:59 04-What is the range of a function Functions Algebra I _ Khan Academy-96uHMcHWD2E.mp4 -rw-------. 1 admin admin 12844 Aug 21 19:55 04-Why heat increases entropy-cKAF1v5hJoE.en.srt -rw-------. 1 admin admin 21284766 Aug 14 13:20 04-Why heat increases entropy-cKAF1v5hJoE.mp4 -rw-------. 1 admin admin 49801130 May 20 07:27 04-World War II in the Pacific in 1942-CBHfoQsF2Y4.mp4 $

balta2ar commented 9 years ago

Thank you for the report! I've reproduced the behavior. The interesting part of the log is:

[download] Destination: ./AP_Macroeconomics-_Challenging_Concepts/03-Classical_and_Keynesian_Models/03-DVDDXN02T314-G060207-iNZ-Viu7U-I.mp4
[download] 100% of 47.64MiB in 01:37
[download] https://www.youtube.com/user/khanacademy => ./AP_Macroeconomics-_Challenging_Concepts/03-Classical_and_Keynesian_Models/04-%(title)s-%(id)s.%(ext)s
Downloading video with URL https://www.youtube.com/user/khanacademy from YouTube.
[youtube:user] khanacademy: Downloading channel page
[youtube:playlist] UU4a-Gbdw7vOaccHmFo40b9g: Downloading webpage
[download] Downloading playlist: Uploads from Khan Academy
[youtube:playlist] UU4a-Gbdw7vOaccHmFo40b9g: Downloading page #1
[youtube:playlist] UU4a-Gbdw7vOaccHmFo40b9g: Downloading page #2
[youtube:playlist] UU4a-Gbdw7vOaccHmFo40b9g: Downloading page #3

I don't think recommendations are related. I think It has found some youtube channels (https://www.youtube.com/user/khanacademy, https://www.youtube.com/user/ACDCLeadership) on the course page in the section of recommended materials and it tried to download them all.

As a temporary solution add this option to your command line: --youtube-dl-options="--flat-playlist". It will do some extra printing, but it won't trash your HDD.

@rbrito @iemejia Guys, what do you think? My first attempt was to use "--no-playlist", but it didn't work, youtube-dl still downloaded tons of videos when it was given user channel URL.

dentarthur commented 9 years ago

Thanks so I have now interrupted at s.11 Money Creation with 5 more sections to go. 1,667 items, totalling 29.3 GB.

I am assuming I should not try temporary solution: --youtube-dl-options="--flat-playlist" as it does not work for others. Will wait for progress.

PS Looks like quite a useful collection from Khan Acadamy!