Closed benoit74 closed 1 year ago
Hey @benoit74, Yeah this all make sense, I can tackle most of this early next week. Thanks for the _python-bootstrap repo link, I'll try and line this repo up with it.
@benoit74 #12 is ready for review, although it doesn't address all the enhancements, it DRY's up the CLI arguments, and tries to match up to other python projects. It also should address #13
I removed the "add support for stats JSON file" part because scraper is running so fast that it makes little sense to report progress.
The scraper needs to be adapted to match openzim conventions.
List below is maybe not yet exhaustive, this is what came into my mind while working on https://github.com/openzim/zimfarm/issues/804.
fix some issues around CLI parameters ; among others probably:
outpath
CLI parameter is not taken into accountoutzim
is not relative tooutpath
and fallbacking toname
, and does not support "{period}" placeholderoutpath
add support for stats JSON fileinstead of multiple commands (
fetch
,prebuild
,zim
), have multiple phases in a single command, and flags to disable some phases if needed ; this will force to have more coherency among parameters of the various phasestake into account new openZIM Python conventions (see https://github.com/openzim/_python-bootstrap, including wiki pages)
PS: @mdp : no worries, this is something we will do on our own (or help you with) ; this is usual on new scrapers and mostly linked to a WIP on our side to better explain our expectations (or not, and consider this issue is normal / to do on our side since very specific to our way of working)