[dev] Docs-site memory optimization

[zFernand0]

These are some suggestions that came out as a result of discussions on how the docs-site could optimize its memory usage.

• Archive old versions of the doc
  • We could continue to use a link from Netlify for a previous build.
  • "<version> LTS": "https://deploy-preview-2745--zowe-docs-master.netlify.app/<version>/getting-started/overview"
  • "v2.1.x LTS": "https://deploy-preview-2745--zowe-docs-master.netlify.app/v2.1.x/getting-started/overview"
  • Note: we may need to update these URLs every so often since Netlify may not keep them around forever.
• Remove all versions that are not actively maintained (in the versioned_docs folder)
  • That include v1.27.x and older, as well as v2.0.x < stable (a.k.a. v2.0.x, v2.1.x, ..., v2.6.x)
• Compress all PDF files
  • Using command line tools for testing purposed, we could these results: (50% reduction)

    $ ps2pdf -dPDFSETTINGS=/ebook zowe-docs-v1.16.x.pdf zowe-docs-v1.16.x-compressed.pdf
    $ ls -al zowe-docs-v1.16.*
    -rw-r--r-- 1 root root  6190459 Mar 24 14:08 zowe-docs-v1.16.x-compressed.pdf
    -rw-r--r-- 1 root root 12308059 Mar 24 14:25 zowe-docs-v1.16.x.pdf

• Videos:
  • Videos with no sound should be converted to GIFs
  • Move all remaining videos to the OMP Youtube channel
• Compress all images (GIFs included)
  • Note: we should not reduce the quality of the images on every Zowe Release.
  • We should implement an automatic image compression for new images introduced in a new Zowe Release
  • Example script provided by @adam-wolfe

    from pathlib import Path
    import os
    for path in Path ('./versioned_docs').rglob('*.gif'):
     print('Optimizing: ' + str(path.absolute()))
     os.system('gifsicle --threads=4 -O3 --colors=256 -o \"' + str(path.absolute()) + '\" \"' + str(path.absolute()) + '\"')

    ---

Benefits After implementing all of these suggestions, we could see the following benefits:

• Reduction of the repository size by more than 50%
• No more out-of-memory errors on Netlify builds
• Faster git-related operations

[1000TurquoisePogs]

note on jpgs & pngs - webms are supported on everything these days, and are generally smaller so it can be a relatively automated process to do it on every image that comes in. it's lossy, but hopefully not an issue.

[1000TurquoisePogs]

I did research on this over the weekend and found many complaints about docusaurus from other users. Simply, it scales badly with quantity of markdown files.

There are things you can do to buy yourself time. As seen in https://github.com/facebook/docusaurus/issues/4765 You can:

• Update to a newer version of docusaurus. We're running a 2 year old beta. I managed to update it to a newer beta before hitting breaking changes that I couldn't find documentation on so I stopped bothering for now.
• Use a different loader, which may use more or less memory, https://blog.johnnyreilly.com/2022/09/29/faster-docusaurus-build-swc-loader
• Build with less concurrency=less memory, but longer builds. This env var controlls it, DOCUSAURUS_SSR_CONCURRENCY. Such as, putting DOCUSAURUS_SSR_CONCURRENCY=4 on this line https://github.com/zowe/docs-site/blob/master/package.json#L8
• Have no versioning at all, or only the latest version of a major version. Only build the latest version. When you think about it, when we release a new version, the last version changes its URLs from /current to /version123 ... so, unless there are bugfixes needed, the content doesn't need to be rebuilt, simply the webserver that's hosting the content needs to change all references from /current to /version123 for that version upon releasing a new version. When it comes to bugfixes on older docs, since our policy is that we make fixes on a major version by applying minor&patch fixes, then this policy can apply to our documentation too such that a typo in 2.4, 2.5, 2.6 can be fixed in 2.current and that be good enough. That would prevent rebuilding.
  • This then requires figuring out how to get the webserver that hosts our content to pick up a change in URLs for the old version.
  • The advantage also being that we can host every single version ever, since it does not effect build. So, that issue with archive versions having their URLs break bookmarks will cease to happen

The above will fix your builds, perhaps indefinitely, but as they're band-aids to the problem, also consider:

• Stop using docusaurus. I'm betting the only thing people care about in our documentation is the ability to quickly find relevant, formatted text. That's been technically possible since the dawn of web browsers, so I'm sure we can find alternatives that don't require tens of gigabytes to build markdown files into websites. There are direct competitors to docusaurus like vuepress, for example. I don't see anything special about our use of docusaurus that would force us to into a corner between having a website at all or not depending on how much ram it consumes. The value of this repo is the knowledge within the markdown files and that's portable to other website builders.

[t1m0thyj]

~I did some research on this last night and it seems we may be able to reduce memory usage by changing our images to be static assets.~

Tested that in this PR and it got past the Out of Memory error. I think this could buy us a lot of time, since our text content is way smaller than images. The main concern I have about this approach is whether performance would suffer for users browsing the site (see https://docusaurus.io/docs/markdown-features/assets#static-assets).

Edit: After further testing, it seems there were other factors involved which tainted initial testing, and changing images to be static would actually increase our memory usage rather than decreasing it.

[KUGDev]

Just some proposals in here. I've looked through the docs directory. There is a dir called "versioned_docs" that is 2 GB approx. Maybe it would be cool to make some versioning based on diffs (something like the GitHub has) instead of carrying all of the versions as is? From my pov, it will reduce the amount of the used space. The second proposal: move all the media (GIFs, images, videos) away from the repo and in the docs repo just bring references to that media files if it is possible. The third one: restrict somehow a size of media files to be no more than some reasonable size (10-20-30 MB). Probably, I can make some research on the proposals, or at least help someone who is interested.

[t1m0thyj]

Update to a newer version of docusaurus. We're running a 2 year old beta. I managed to update it to a newer beta before hitting breaking changes that I couldn't find documentation on so I stopped bothering for now.

Made an attempt at this in the test-docusaurus-update-timothy branch if anyone is curious to investigate.

~Got pretty far in the process of upgrading from v2.0.0-beta to the latest version (currently v2.4.0). Had to add some Webpack polyfills and update some Docusaurus imports, but after doing that the latest error has me kind of stuck. Every page fails to build with this error: TypeError: Cannot read properties of undefined (reading 'label')~

Update: Thanks @traeok for fixing the build to work with 2.4.0: https://github.com/zowe/docs-site/pull/2865