jcline-ieee
commented
6 years ago I have partially implemented this and patch included here [for 0.1.6]. The "text comments" which are normally below the video on a web page are now saved for each course Unit, into a local Section web page named "Notes-[Section title].html". This is very handy for seeing those instructor comments, sometimes homework tips, and also where other resources are used (like the PDF's). The output has limitations and isn't super pretty but the edx course text is there. Embedded graphics from the page are not fetched to local so that is still a todo (but maybe those should be added to the 'resources' download list). The result is a tree of files like this:
01-Important_Pre-Course_Survey/
02-Contact_Us/
03-How_To_Navigate_the_Course/
04-Discussion_Board/
05-Office_Hours/
06-Week_1-__Introduction_to_Data/
07-Week_2-__Univariate_Descriptive_Statistics/
08-Week_3-__Bivariate_Distributions/
09-Week_4-__Bivariate_Distributions_Categorical_Data/
10-Week_5-__Linear_Functions/
11-Week_6-__Exponential_and_Logistic_Function_Models/
12-Important_Post-Course_Survey/
Notes-01-Important_Pre-Course_Survey.html
Notes-02-Contact_Us.html
Notes-03-How_To_Navigate_the_Course.html
Notes-04-Discussion_Board.html
Notes-05-Office_Hours.html
Notes-06-Week_1-__Introduction_to_Data.html
Notes-07-Week_2-__Univariate_Descriptive_Statistics.html
Notes-08-Week_3-__Bivariate_Distributions.html
Notes-09-Week_4-__Bivariate_Distributions_Categorical_Data.html
Notes-10-Week_5-__Linear_Functions.html
Notes-11-Week_6-__Exponential_and_Logistic_Function_Models.html
Notes-12-Important_Post-Course_Survey.html I tested on macOs python 3.6 with a few small courses and a couple big ones, like: https://courses.edx.org/courses/course-v1:UTAustinX+UT.7.21x+3T2016/course/ Seems to work very well. Unfortunately the output page contains extraneous html but the course content is saved and readable. There are important course notes saved now, that I would not have otherwise seen.
See below PDF for what the HTML output result looks like. Foundations_of_DataAnalysis-_Part_1 - Week 1: Introduction to Data>.pdf
Warning. The "page dump" to the local HTML file is not filtered when saving and so it will contain HTML tags and/or bits of HTML script which have identifying information of the original edx user account (like user name, user id). So the result is not 'private'.
Implementation Detail
- Add 'doc' to the Unit object
- when extracting units in parsing, save the 'text' to the Unit.doc
- in main, before filtering out duplicate urls, loop thru the course's Units and save the Unit text from Unit.doc and output to HTML files per section at the top level. Therefore each "weekly lesson" directory has a corresponding HTML page including the notes for those Units.
- Add some customized HTML markup to the HTML file, like page title, and to make headers for the Unit, and also list the resources, for easy cross-referencing.
Patch
This patch will need cleanup and refactoring. But it works. To apply the patch add or modify the stuff labelled 'jcline'. Sorry no real patch diff.
edx_dl.py
import re
import sys
from six.moves import html_parser # jcline
....
def main():
....
parse_units(selections)
if args.cache:
write_units_to_cache(all_units)
# jcline
# -- Write each edx html page content to Notes-[SECTION NAME].html
for selected_course, selected_sections in selections.items():
coursename = directory_name(selected_course.name)
for selected_section in selected_sections:
section_dirname = "%02d-%s" % (selected_section.position,
selected_section.name)
target_dir = os.path.join(args.output_dir, coursename)
mkdir_p(target_dir)
counter = 0
filename = os.path.join(target_dir, "Notes-%s.html" % clean_filename(section_dirname))
logging.info("Writing Course Section document to %s", filename)
with open(filename, 'w') as f:
markup = "<!DOCTYPE html>\n<html><head>\n"
markup += "<title>%s - %s</title>\n" % (coursename, selected_section.name)
markup += "</head>\n"
markup += "<body><h1>%s</h1><h1>%s</h1>\n" % (coursename, selected_section.name)
f.write(markup)
for subsection in selected_section.subsections:
units = all_units.get(subsection.url, [])
for unit in units:
counter += 1
markup = "\n<h2> Unit %d </h2>\n" % counter
f.write(markup)
f.write(html_parser.HTMLParser().unescape(unit.doc))
markup = "\n<h2> Media </h2>\n<ul>\n"
for video in unit.videos:
markup += "<li>" + " ".join(video.mp4_urls) + "\n"
for item in unit.resources_urls:
markup += "<li>" + item + "\n"
markup += "\n</ul>\n"
f.write(markup)
markup = "\n\n<p><footer>Created by edx_dl %s </footer>\n<!-- __END__ -->\n</html>\n" % __version__
f.write(markup)
# --- jcline
# This removes all repeated important urls
# FIXME: This is not the best way to do it but it is the simplest, a
# better approach will be to create symbolic or hard links for the repeated
# units to avoid losing information
filtered_units = remove_repeated_urls(all_units)
num_all_urls = num_urls_in_units_dict(all_units)parsing.py
class CurrentEdXPageExtractor(ClassicEdXPageExtractor):
...
def extract_unit(self, text, BASE_URL, file_formats):
....
resources_urls = self.extract_resources_urls(text, BASE_URL,
file_formats)
return Unit(videos=videos, resources_urls=resources_urls, doc=text) # jclinecommon.py
class Unit(object):
....
self.videos = videos
self.resources_urls = resources_urls
self.doc = doc # jcline
rbrito
TheNameIsChaitanya
nikojpapa
Dinesh6777
🚨Please review the Troubleshooting section before reporting any issue. Don't forget also to check the current issues to avoid duplicates.
Subject of the issue
Describe your issue here. I would be greatful if someone can add an additional command line argument say --pg which will enable edx-dl to save all the webpages for a particular edx course in a seperate folder within the course folder. I really need someone to help me on this. because from where I come from internet connections are not really stable and I need to be able to view this materials independent of internet connectivity. Thanks in advance
Your environment
Steps to reproduce
Tell us how to reproduce this issue. Please provide us the course URL, and the specific subsection or unit if possible.
Expected behaviour
Tell us what should happen.
Actual behaviour
Tell us what happens instead. If the script fails, please copy the entire output of the command or the stacktrace (don't forget to obfuscate your username and password). If you cannot copy the exception, attach a screenshot.