Create the initial implementation

[josephlewis42]

Create the initial implementation of the new Devdocs scraper as requested in https://github.com/openzim/zim-requests/issues/1086.

Once implemented and merged, use the knowledge gained to complete https://wiki.openzim.org/w/index.php?title=How-to_create_a_Python_scraper&gettingStartedReturn=true

[josephlewis42]

I've been working on this in the background poking at various ways forward. A couple of the major things I've found are:

• Initially it seemed like the list of docs was only known server-side, but I've since found what appears to be a (potentially) unreferenced JSON file that's generated by devdocs and can be fetched using requests
• Devdocs does a lot of pre-compilation for SCSS assets during their docker build pipeline. This includes logos for the project which are in the git repo but not as published artifacts which means the ZIM cover photos are likely going to be generic unless we also check out the git repo.
• Each project has CSS overrides to help standardize the docs. We probably need to bring these along
• Some of the logic for rendering document trees is only client-side and done in a kludgy way with regex.

My plan of attack is this:

1. [x] Get the type definitions and devdocs HTTP client in with tests.
2. [ ] Add the skeleton CLI that doesn't generate a ZIM.
3. [ ] Generate a ZIM using Python templates. The ZIM's search index will replace the devdocs sidebar index.

Once that's done we'll have a working MVP and I can look at building out a more modern front-end.

[benoit74]

Plan seems to be appropriate, nothing much to add but some minor remarks:

• I like the MVP approach, it looks like only the MVP is going to be a significant work and it is always good to see progress.
• We can checkout the git repo if it add value ; this is done in freecodecamp with the "zip" strategy: https://github.com/openzim/freecodecamp/blob/a1221073049895609d7712a8834474311535f0c1/scraper/src/fcc2zim/fetch.py#L12
• Having proper cover photos is probably a nice to have, we can live without it in a first version
• Good luck with CSS overrides and kludgy regex ; do we need to reimplement it on our side or can we automatically reuse the code from their git repo? If we reimplement it on our side, it is mandatory to mention where this code comes from (which file(s)) and what is the commit / date when the "sync" was done, so that maintenance is possible (we will probably need to update this code at some point)

[josephlewis42]

do we need to reimplement it on our side or can we automatically reuse the code from their git repo? If we reimplement it on our side, it is mandatory to mention where this code comes from (which file(s)) and what is the commit / date when the "sync" was done, so that maintenance is possible (we will probably need to update this code at some point)

It looks to me like the JS is too wrapped up with features we wouldn't want (e.g. service workers, local database usage to "download" items for offline use, SPA, click tracking) and they'd cause a confusing overlap with the functionality of ZIM readers so pulling bits out will likely be difficult.

In this specific case, the code has been the same for over eight years so I anticipate it won't change much in the future. However, I'll be sure to leave notes.

I haven't talked with the maintainers over there, but there might be opportunities to help them and us at the same time either by creating a front-end that can optionally exclude the bits we don't want so we could package the app wholesale or by pushing some of this frontend logic into the data itself.

[josephlewis42]

I've been able to make some significant progress on this locally with Jinja to get us close to the stated MVP which I'll split into PRs:

A couple of limitations I'm noticing that I want to document as things to follow up on:

• All docs are in light mode but adding dark mode based on user preference should be simple with a little JS.
• Syntax highlighting isn't currently working because it's done client side.
• The sidebar padding is a little off because I'm using <details> rather than JS.

[benoit74]

Woah, good job, looks impressive so far!