search

coursera-dl / edx-dl

A simple tool to download video lectures from edx.org (and other openedx sites)
GNU Lesser General Public License v3.0
1.93k stars 639 forks source link

edx-dl downloads incomplete/duplicate content with different name #249

Open shockme opened 9 years ago

shockme commented 9 years ago

OS: Ubuntu 14.04 Python 2.7.6 youtube-dl 2015.06.25 course URL: https://courses.edx.org/courses/course-v1:IEEEx+CloudIntro.x+2015T2/info

For this course (I haven't tested for others) on the first run edx-dl doesn't download all the videos, and from the 2nd run onwards it starts duplicating content.

This is the output of diff -y -W 200 ../run1_files.txt ../run2_files.txt (you need to scroll sideways a lot :( ) :

Introduction_to_Cloud_Computing                                                                         Introduction_to_Cloud_Computing
├── 01-Course_Introduction_and_Overview                                                                 ├── 01-Course_Introduction_and_Overview
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.en.srt                                      │   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.en.srt
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.mp4                                         │   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.mp4
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.en.srt                                                    │   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.en.srt
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.mp4                                                       │   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.mp4
│   ├── 03-edX – How It Works-3_yD_cEKoCk.en.srt                                                        │   ├── 03-edX – How It Works-3_yD_cEKoCk.en.srt
│   └── 03-edX – How It Works-3_yD_cEKoCk.mp4                                                           │   └── 03-edX – How It Works-3_yD_cEKoCk.mp4
├── 02-Part_1-_Overview_of_Cloud_Computing                                                              ├── 02-Part_1-_Overview_of_Cloud_Computing
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.en.srt                                              │   ├── 01-01 1 What is Cloud Computing-onYDs37125g.en.srt
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.mp4                                                 │   ├── 01-01 1 What is Cloud Computing-onYDs37125g.mp4
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.en.srt            │   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.en.srt
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.mp4               │   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.mp4
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.en.srt                                               │   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.en.srt
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.mp4                                                  │   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.mp4
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.en.srt                          │   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.en.srt
│   └── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.mp4                        |    │   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.mp4
                                                                                                   >    │   ├── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.en.srt
                                                                                                   >    │   └── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.mp4
├── 03-Part_2-_Dynamic_Interactions_and_Computing_Architectures                                         ├── 03-Part_2-_Dynamic_Interactions_and_Computing_Architectures
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.en.srt                                          │   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.en.srt
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.mp4                                             │   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.mp4
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.en.srt                             │   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.en.srt
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.mp4                                │   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.mp4
                                                                                                   >    │   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.en.srt
                                                                                                   >    │   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.mp4
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1sw        │   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1sw
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1sw        │   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1sw
                                                                                                   >    │   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
                                                                                                   >    │   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt                            │   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4                               │   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
                                                                                                   >    │   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.sr
                                                                                                   >    │   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUr        │   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUr
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUr        │   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUr
                                                                                                   >    │   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.en.srt
                                                                                                   >    │   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.sr        │   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.sr
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4          │   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
                                                                                                   >    │   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
                                                                                                   >    │   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.en.srt                                                      │   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.mp4                                                         │   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.mp4
                                                                                                   >    │   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
                                                                                                   >    │   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt                                                       │   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.mp4                                                          │   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
                                                                                                   >    │   ├── 08-02 10 Summary-v-fF8NAzAjg.en.srt
                                                                                                   >    │   ├── 08-02 10 Summary-v-fF8NAzAjg.mp4
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt                              │   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4                                 │   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 10-02 10 Summary-v-fF8NAzAjg.en.srt                                                             │   ├── 10-02 10 Summary-v-fF8NAzAjg.en.srt
│   └── 10-02 10 Summary-v-fF8NAzAjg.mp4                                                                │   └── 10-02 10 Summary-v-fF8NAzAjg.mp4
shockme commented 9 years ago

$ cat run1_files.txt

Introduction_to_Cloud_Computing
├── 01-Course_Introduction_and_Overview
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.en.srt
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.mp4
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.en.srt
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.mp4
│   ├── 03-edX – How It Works-3_yD_cEKoCk.en.srt
│   └── 03-edX – How It Works-3_yD_cEKoCk.mp4
├── 02-Part_1-_Overview_of_Cloud_Computing
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.en.srt
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.mp4
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.en.srt
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.mp4
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.en.srt
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.mp4
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.en.srt
│   └── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.mp4
├── 03-Part_2-_Dynamic_Interactions_and_Computing_Architectures
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.en.srt
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.mp4
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.en.srt
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.mp4
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.en.srt
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.mp4
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.en.srt
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.mp4
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 10-02 10 Summary-v-fF8NAzAjg.en.srt
│   └── 10-02 10 Summary-v-fF8NAzAjg.mp4
├── 04-Part_3-_Economics_of_Cloud_Computing
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.en.srt
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.mp4
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.en.srt
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.mp4
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.en.srt
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.mp4
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.en.srt
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.mp4
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.en.srt
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.mp4
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.en.srt
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.mp4
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.en.srt
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.mp4
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.en.srt
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.mp4
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.en.srt
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.mp4
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.en.srt
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.mp4
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.en.srt
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.mp4
│   ├── 12-03 11 Part 3 Summary-J1euyoxs4sM.en.srt
│   └── 12-03 11 Part 3 Summary-J1euyoxs4sM.mp4
└── 05-Part_4-_Conclusion
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.en.srt
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.mp4
    ├── 02-Cloud_Computing_Synopsis.pdf
    ├── 02-imc10-cloudcmp.pdf
    ├── 02-mco2012060048.pdf
    ├── 02-SP800-144.pdf
    └── 02-SP800-145.pdf

5 directories, 65 files
shockme commented 9 years ago

$ cat run2_files.txt

Introduction_to_Cloud_Computing
├── 01-Course_Introduction_and_Overview
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.en.srt
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.mp4
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.en.srt
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.mp4
│   ├── 03-edX – How It Works-3_yD_cEKoCk.en.srt
│   └── 03-edX – How It Works-3_yD_cEKoCk.mp4
├── 02-Part_1-_Overview_of_Cloud_Computing
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.en.srt
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.mp4
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.en.srt
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.mp4
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.en.srt
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.mp4
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.en.srt
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.mp4
│   ├── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.en.srt
│   └── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.mp4
├── 03-Part_2-_Dynamic_Interactions_and_Computing_Architectures
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.en.srt
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.mp4
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.en.srt
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.mp4
│   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.en.srt
│   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.mp4
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.en.srt
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.mp4
│   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.en.srt
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.mp4
│   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 08-02 10 Summary-v-fF8NAzAjg.en.srt
│   ├── 08-02 10 Summary-v-fF8NAzAjg.mp4
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 10-02 10 Summary-v-fF8NAzAjg.en.srt
│   └── 10-02 10 Summary-v-fF8NAzAjg.mp4
├── 04-Part_3-_Economics_of_Cloud_Computing
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.en.srt
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.mp4
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.en.srt
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.mp4
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.en.srt
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.mp4
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.en.srt
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.mp4
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.en.srt
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.mp4
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.en.srt
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.mp4
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.en.srt
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.mp4
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.en.srt
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.mp4
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.en.srt
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.mp4
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.en.srt
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.mp4
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.en.srt
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.mp4
│   ├── 12-03 11 Part 3 Summary-J1euyoxs4sM.en.srt
│   └── 12-03 11 Part 3 Summary-J1euyoxs4sM.mp4
└── 05-Part_4-_Conclusion
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.en.srt
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.mp4
    ├── 02-Cloud_Computing_Synopsis.pdf
    ├── 02-imc10-cloudcmp.pdf
    ├── 02-mco2012060048.pdf
    ├── 02-SP800-144.pdf
    └── 02-SP800-145.pdf

5 directories, 81 files
shockme commented 9 years ago

$ cat run3_files.txt

Introduction_to_Cloud_Computing
├── 01-Course_Introduction_and_Overview
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.en.srt
│   ├── 01-IEEE Introduction to Cloud Computing-RSpX21RFGDM.mp4
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.en.srt
│   ├── 02-CourseOverviewLaplante-_lzJlf_CSgA.mp4
│   ├── 03-edX – How It Works-3_yD_cEKoCk.en.srt
│   └── 03-edX – How It Works-3_yD_cEKoCk.mp4
├── 02-Part_1-_Overview_of_Cloud_Computing
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.en.srt
│   ├── 01-01 1 What is Cloud Computing-onYDs37125g.mp4
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.en.srt
│   ├── 02-01 2 Service and Deployment, Infrastructure, and Consumer View-i9Kj_ccf5GM.mp4
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.en.srt
│   ├── 03-01 3 Security and Scenarios-HmnSiS7gCWc.mp4
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.en.srt
│   ├── 04-01 4 Assumptions, Terms of Service, and Promises-9kzjLBQijpg.mp4
│   ├── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.en.srt
│   └── 05-01 5  Limitations, Obligations, Recommendations, Implications-5w7J-b3PM7A.mp4
├── 03-Part_2-_Dynamic_Interactions_and_Computing_Architectures
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.en.srt
│   ├── 01-02 00 Part 2 Instructor Overview-uHGE3Tt3G7w.mp4
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.en.srt
│   ├── 02-02 01 Service, Deployment, Scope, and Control-Vk0eU1kBEiY.mp4
│   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.en.srt
│   ├── 02-02 02 SaaS Dynamics and Software Stack Control-tsTVLAHKfyY.mp4
│   ├── 02-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.en.srt
│   ├── 02-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.mp4
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.en.srt
│   ├── 03-02 03 SaaS Benefits, Issues and Concerns, Suitability, and Recommendations--28nwkU1swo.mp4
│   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 03-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.en.srt
│   ├── 04-02 04 PaaS Dynamics and Software Stack Control-4dkkeJUvKws.mp4
│   ├── 04-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.en.srt
│   ├── 04-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.mp4
│   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 04-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.en.srt
│   ├── 05-02 05 PaaS Benefits, Issues and Concerns, Suitability, and Recommendations-NkXrtJLzUro.mp4
│   ├── 05-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 05-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 05-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.en.srt
│   ├── 06-02 06 IaaS Abstract Interaction Dynamics and Software Stack Control-z5DAwqkMZMA.mp4
│   ├── 06-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 06-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 06-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.en.srt
│   ├── 07-02 07 IaaS Operation-wgKraDIPpYk.mp4
│   ├── 07-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 07-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 07-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.en.srt
│   ├── 08-02 08 IaaS Benefits-MNSO6ELqJu8.mp4
│   ├── 08-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 08-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 08-02 10 Summary-v-fF8NAzAjg.en.srt
│   ├── 08-02 10 Summary-v-fF8NAzAjg.mp4
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.en.srt
│   ├── 09-02 09 IaaS Issues and Concerns, and Benefits-6PbOo5EMEX4.mp4
│   ├── 09-02 10 Summary-v-fF8NAzAjg.en.srt
│   ├── 09-02 10 Summary-v-fF8NAzAjg.mp4
│   ├── 10-02 10 Summary-v-fF8NAzAjg.en.srt
│   └── 10-02 10 Summary-v-fF8NAzAjg.mp4
├── 04-Part_3-_Economics_of_Cloud_Computing
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.en.srt
│   ├── 01-03 00 Part 3 Instructor Overview-irQOV7V0OWU.mp4
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.en.srt
│   ├── 02-03 01 Review of Service Models-Y0QPbA8wH2E.mp4
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.en.srt
│   ├── 03-03 02 SWOT Analysis and Value Proposition-It3PcIh73HE.mp4
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.en.srt
│   ├── 04-03 03 General Cloud Computing Risks-otP0NmXMZo8.mp4
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.en.srt
│   ├── 05-03 04 Risks 2 Performance, Network Dependence, Reliability, Outages, and Safety Critical Processing-2jbkrYUU5JY.mp4
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.en.srt
│   ├── 06-03 05 Risks 3 Compliance and Information Security-KcYd2RLqC48.mp4
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.en.srt
│   ├── 07-03 06 Value and Risk of Open Source Software-y9ePOrsFhRI.mp4
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.en.srt
│   ├── 08-03 07 Cloud Computing Cost Analysis-7r4yl5dk91Q.mp4
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.en.srt
│   ├── 09-03 08 Selecting an IaaS Provider-r93FbHA54K0.mp4
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.en.srt
│   ├── 10-03 09 Cloud Standards and Intercloud Interoperability-qgzK033V5Hk.mp4
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.en.srt
│   ├── 11-03 10 Recommendations for Successful Cloud Migration-pbza6B7ChTQ.mp4
│   ├── 12-03 11 Part 3 Summary-J1euyoxs4sM.en.srt
│   └── 12-03 11 Part 3 Summary-J1euyoxs4sM.mp4
└── 05-Part_4-_Conclusion
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.en.srt
    ├── 01-04 00 Course Conclusion-AyeAZizjxHA.mp4
    ├── 02-Cloud_Computing_Synopsis.pdf
    ├── 02-imc10-cloudcmp.pdf
    ├── 02-mco2012060048.pdf
    ├── 02-SP800-144.pdf
    └── 02-SP800-145.pdf

5 directories, 95 files
rbrito commented 9 years ago

@shockme, yes, I have run into that myself too.

As a workaround, please try running the script with the option --sequence. I will, BTW, change that name for our (already delayed) first release so that it is called --sequential instead.

Or, perhaps, even make it the default, while the multithreaded version being set as marked "experimental" or something like that.

Please, report back your experience with the change in the behavior.

Thanks,

Rogério.

iemejia commented 9 years ago

I don't think that the sequential extraction fixes this, or I don't see the logic of why it would do it, if there are three independent processes:

  1. extracting resources (getting the 'downloadable' URLs)
  2. removing repeated resources (delete extra URLs)
  3. download the URLs

The only process that happens in parallel is 1, so I can't see any relation between sequential/parallel parsing and the bug, but maybe there is something I am missing here.

rbrito commented 9 years ago

@iemejia, the thing is that I'm not getting a reproducible behavior with the script. Sometimes I do, sometimes, I don't.

I have not paid too much attention to how we extract resources (I will do that in the near future, I hope :)), but we are surely missing some stuff. Also, the deduplication phase is not working as it should. I guess that pickling/printing the stuff that we get for post-mortem analysis is the way to go here.

kirill-gr commented 8 years ago

It may be better to build the dict of files for downloading (keyed by url), then download all of them and finally sort by section folder, making multiple copies only if needed (for example, it could make sense to have the same file appearing in different sections, but it also may be a subject of preferences). This should eliminate unnecessary duplication.

iemejia commented 8 years ago

@kromkrom I implemented a similar idea in the remove_repeated_urls function (in edx_dl.py), I created a set of urls (that corresponds to the keyed dict you mention, and I traverse the course hierarchy removing the urls that were part of the set, so they won't be downloaded many times. However I never had the time to fully test it and sometime later some errors were reported and it was changed, but I didn't follow the changes so I don't know if the approach used is still the same.

I agree with you about the preferences, I know some people don't want the repeated files at all, but I really would like to have the duplicate videos if not repeated at least created as hard links to keep the consistency of each course (e.g. with videos that exist in multiple sections).

If you have the time (and the courage) to work in this fix, please tell me and maybe we can fix it together.

kirill-gr commented 8 years ago

@iemejia I've submitted a pull request with the draft implementation

k0ala commented 6 years ago

@kromkrom could you point me to your PR?

kirill-gr commented 6 years ago

@k0ala, I believe my PR is completely out-of-date