Draft: First pass at content

[mdp]

First pass at adding content from FCC to project. This is a pared down version which only has the JS courses (~85mb)

The content root is located at: https://github.com/openzim/freecodecamp/tree/mdp/first_content/content/learn/javascript-algorithms-and-data-structures

[kelson42]

@mdp Great! I'm not sure if your PR is ready, but not beeing in draft, I assumed "yes" and have put @rgaudin as a reviewer.

In general, put PRs in draft if not ready to review. Otherwise request review from @rgaudin.

[kelson42]

@mdp No way to retrieve content dynamically from upstream git repository and then apply treatment? Any strong reason to duplicate content in our repository? We don't host any content here usually, worse case in a tarball stored externally.

[rgaudin]

Hi @mdp, thanks for the first PR.

Can you fill me in on what you're trying to achieve? It's not clear to me at this point. I understand the ultimate goal being to have a script/tool to generate a ZIM file for FCC though! 😅

As @kelson42 said, we only host original code (or content) in our repo unless we abvolutely have to. I see that the code ressemble the markdown files at https://github.com/freeCodeCamp/freeCodeCamp/tree/main/curriculum/challenges/english/02-javascript-algorithms-and-data-structures

• Is this compiled form available online? If so, we should download it.
• If not, would it be possible to use a script to fetch from upstream and convert it?

It would have the additional benefit of being somewhat update-able while this is not.

Let me know what you think and if you have questions.

Few links for the following step:

• Our Scrapers list
• libzim (C++)
• python-libzim
• scraperlib (has a wrapper around python-libzim)
• node-libzim
• zimwriterfs (binaries) builds a ZIM file from an offline-ready filesystem tree. scraperlib also bundles a similar feature (zim.filesystem.make_zim_file)

Also, without having looked at this content, I stumbled upon a cats.json file that references online content. No idea how/if this is gonna be used but of course this would not work offline.

"imageLink": "https://s3.amazonaws.com/freecodecamp/funny-cat.jpg",`

[kelson42]

@rgaudin I have explicitly recommended to @mdp, not to use zimwriterfs.

[mdp]

Yeah, sorry for the big dump of files. Yes, ideally I'd like to have the following happen in some type of CI system.

• Download the latest FCC curriculum and build it with our custom UI changes (removing login and some changes to the pages)
• Package up the content of a particular course into a zim format using one of the above tools.
• Upload that content to wherever it needs to go.

Some concerns:

• We had issues getting this into a CI pipeline before because of the size of the project and time it took to build, but it's doable once you get the dependencies cached. However it still might be a concern.
• Is there any way we can pull the content from something like an S3 bucket and I could place the content into that bucket and have this repository package it up?
• How do we want to build the course curriculum in the future to keep it up to date? Unfortunately it's a somewhat manual process at the moment. We have our own version of FCC which build the pages in a particular way using the standard curriculum data (the Markdown files). I think the previous implementation was to cherry pick the FCC curriculum directly from their repository then build it using our customized version.

For this first pass, I'm going to create a fcc2zim script to pull the data from a specified folder of content, and package this up into a zim format for testing. Instead of checking the course content into this repo, I'll use a local copy which I can provide for others wanting to test it. I've removed the content from this repository.

[rgaudin]

OK ; we usually don't create ZIM files in CI because it adds too many constraints: resource, readability, etc.

What we do for similar projects would be to download an archive of the content to convert from our shared drive. Then scripts stored in this repo would download it and build the ZIM file.

Then we have two scenarios:

• it gets updated frequently: we create a docker image for that script and it gets called by the zimfarm (and uploaded automatically)
• it's not ; then it is run manually and uploaded once.

I believe we're talking about the latter ; in this case, a script(s) with proper instructions in the README are probably the way to go. See lilote for instance.

Let me know once you have an archive of the content so I can upload it to the drive.

[kelson42]

OK ; we usually don't create ZIM files in CI because it adds too many constraints: resource, readability, etc.

We have our own toolchain for that, look at https://farm.openzim.org

Typicall workflow is:

• Build the scraper
• Publish Docker image of it
• Add support (code driven) of this specific new scraper in Zimfarm
• Create specific recipes (data driven, here we will have only one recipe I guess)
• Run the recip to generate ZIM in the ZIM farm which distribute the work and upload the ZIM to http://download.kiwix.org/zim

[mdp]

Closing in favor of #5