search

openzim / openedx

Open edX (to zim) scraper
GNU General Public License v3.0
8 stars 7 forks source link

Prevent duplicate files #108

Closed satyamtg closed 4 years ago

satyamtg commented 4 years ago

This fixes #104 by downloading everything to instance_assets instead of downloading to different places. Also prevents creation of dummy subtitles if content was {}, which resulted in many duplicates in zimcheck

Please do note that this prevents duplicate files on the basis of unique URLs, however there are some small number of files which have different URLs but same content, and hence are reported by zimcheck. There's no way of identifying them without calculating a checksum. I think we shall see https://github.com/openzim/python_scraperlib/issues/33 and use that instead for a small subset of files (which are basically some images under instance_assets).

satyamtg commented 4 years ago

Should we add a function that returns the "../" * x? Might bit a bit more readable?

Have added a function get_back_jumps(nb_jumps) in utils.py