Merge edx-downloader with coursera-dl or create mooc-dl

[balta2ar]

Hello. I'm probably not the first one to come up with this idea but to me it seems to be a very logical next step. There are two distinct projects that do basically the same but on different MOOC platforms. Support for these platforms sounds like a good code modularization challenge, it would also be a good programming style. coursera-dl looks more advanced in terms of different kind of optimizations like cookie management, caching and so on, so it's edx-downloader to merge with coursera-dl, not vise versa.

I would like to know what you think? Is this reasonable? If no, why? If yes, what steps can be taken to start moving in this direction?

[shk3]

Thanks for your suggestion. It is indeed a brilliant idea. Most of the collaborators of this project are currently busy for life or study. We have made a goal to refactorize the project to make it robust, but we have not archived it yet. So, I think we probably cannot realize this enhancement at this time, since much more efforts of integrating different projects will be involved. We will try our best to finish our current goal, and maybe realize this feature in the future.

[balta2ar]

I see some inspiring activity in this repo lately. Has situation changed since my request was created? @shk3, @rbrito, guys?

[rbrito]

@balta2ar, thanks for the reminder. Yes, we are slowly refactoring things and some functions/pieces of coursera-dl are already being used here in edx-dl. I hope to have them soon sharing more code (e.g., the non-site specific parts), since life is so short. :)

Oh, I would love to have more contributors both here or on coursera-dl. Any help (even triaging bug reports, seeing if a bug is still present or not, etc.) is welcome.

[balta2ar]

@rbrito @shk3 Thinking about it a little more... What do you think about merging it with youtube-dl? After all, all these tools are made for tricky extraction of file links from the sites and downloading them. Plus, youtube-dl already has a great infrastructure and it's easy to add new sources. Is this too crazy idea, why, and why not?

[iemejia]

I will continue the discussion here from the previous mention in coursera-dl.

The best approach i think we can do is incremental, start moving common stuff. I think we can create first mooc-dl as a commons for the shared functions/methods, and refactor both projects with mooc-dl as a dependency and then move step by step to share as much code as possible until both projects fusion.

This will be probably the first file I will put there (and its equivalent in coursera-dl): https://github.com/coursera-dl/edx-dl/blob/master/edx_dl/utils.py

@rbrito Can you create the repo so we can start working there, and @balta2ar once created can you put the equivalent parts of coursera-dl there (the utils) and publish mooc-dl in pip, like this I will take it as an edx-dl dependency and start moving the deps there too.

[balta2ar]

Sure. Speaking of pip packages, @rbrito @iemejia, could you give me your PyPI user names (can be found here: https://pypi.python.org/pypi?%3Aaction=user_form) so that I could add you both to owners of coursera-dl, edx-dl and mooc-dl, and we all could do PyPI releases?

[jcline-ieee]

why is edx-dl not made into a youtube-dl downloader? it uses youtube-dl extensively anyway.

[balta2ar]

I've gotten a bit less enthusiastic about this idea over the years. For starters, youtube-dl is simpler in terms of the structure that it generates. A general use case is to extract one file and save it. In edx-dl it's never one file, it's always a course with many files. The second reason is of course lack of free time. Third, we've seen edx emloyees coming here and reminding us that this script is violating user agreement terms. To say the least, it's not the best encouragement, and it definitely kills the motivation (at least for me).