Jenkins job to test relaxed requirements

[oraNod]

Create a jenkins job to periodically test the relaxed requirements without deploying to the web server. The purpose is to catch dependency issues with the build early.

[webknjaz]

May I suggest using GHA instead? It'd allow the logs to be publicly visible as opposed to being in a hidden location, only accessible by some secret society...

[oraNod]

May I suggest using GHA instead? It'd allow the logs to be publicly visible as opposed to being in a hidden location, only accessible by some secret society...

The only way I've managed to get the package docs to build with GHA is with a private runner that we used for https://github.com/ansible/docsite/commit/abe9c90b4d76050f0005fcd11590b8375489cb11

I realize another job that is RH internal isn't ideal. We could probably also use our Matrix webhook to notify the docs channel on build failure. We could probably export logs somewhere on the web server to increase access/visibility as a temporary measure.

[webknjaz]

The only way I've managed to get the package docs to build with GHA is with a private runner that we used for ansible/docsite@abe9c90

I suppose you mean that rsync can't reach internal deploy targets, right? But from the issue description, I thought that you wanted to check the build, not deployment? Do you actually need to publish those files somewhere?

[oraNod]

The only way I've managed to get the package docs to build with GHA is with a private runner that we used for ansible/docsite@abe9c90

I suppose you mean that rsync can't reach internal deploy targets, right? But from the issue description, I thought that you wanted to check the build, not deployment? Do you actually need to publish those files somewhere?

No, I want to check the package docs build with the relaxed requirements. There's no need to deploy the generated html to the web server.

[webknjaz]

No, I want to check the package docs build with the relaxed requirements. There's no need to deploy the generated html to the web server.

This means that you don't need Jenkins for this check at all.

[oraNod]

No, I want to check the package docs build with the relaxed requirements. There's no need to deploy the generated html to the web server.

This means that you don't need Jenkins for this check at all.

@webknjaz Nice, I guess I'm missing something here. I'd love to build package docs (make webdocs) via GHA or some other way that is more accessible to the community. So far I've run into problems with resource constraints as I mentioned before. Nothing on the "free" tier seemed to be feasible. Maybe I'm a bit fixated though. What do you suggest?

[webknjaz]

I haven't seen that but show me the problem with logs and I'll take a look.

[oraNod]

@webknjaz We were working on this back in Feb / March so all those old logs from the failed jobs are long gone. They also would have applied to the RST in ansible/ansible.

I've refreshed the workflow to use ansible-documentation here: https://github.com/oraNod/ansible-documentation/commit/84ee1ac9616177d5cd5cc3306a3d81d852ae9419

Edit: Update to the right commit and remove link to the broken workflow.

[oraNod]

Ah, no. I've made a mistake in that workflow. It looks like it's still configured to use a self-hosted runner. I'll fix that now.

[oraNod]

Here we go: https://github.com/oraNod/ansible-documentation/actions/runs/6199270037/job/16831441141

[webknjaz]

@oraNod could you try using acitons/cache@v3 against '_build/doctrees'? I wonder if Sphinx would be able to make use of the doctree cache and skip rebuilding whenever possible...

[oraNod]

@webknjaz I've added https://github.com/oraNod/ansible-documentation/commit/37f934ad3882a1dc7aedb7f412fd1e09fd6c6093 but there would be no existing _build/doctrees in this case would there?

[oraNod]

Also that job I kicked off has failed. It either exceeds the max runtime for the default runner or eventually runs out of memory and fails. Here are the logs: https://pipelinesghubeus7.actions.githubusercontent.com/qE2VW7Snvxf8Tyazeylwmc6UFEmrIlPo87phzl7UuLWuXKKanS/_apis/pipelines/1/runs/111/signedlogcontent/2?urlExpires=2023-09-15T15%3A43%3A38.1649893Z&urlSigningMethod=HMACV1&urlSignature=4hW5pSqdLp8C4Oo5v4u%2FhD3w7L7chqcXfyBUygAjmho%3D

I've also kicked off another job using the cache action: https://github.com/oraNod/ansible-documentation/actions/runs/6200113239/job/16834084857

[webknjaz]

I've added oraNod@37f934a

I added some comments there.

but there would be no existing _build/doctrees in this case would there?

After git checkout, no. After downloading the cache — might be. After running Sphinx — definitely.

[webknjaz]

I also discovered that it's necessary to run git-restore-mtime post checkout https://github.com/MestreLion/git-tools/tree/main#git-restore-mtime per https://github.com/sphinx-doc/sphinx/issues/11556.

This snippet https://github.com/sphinx-doc/sphinx/issues/11556#issuecomment-1675988933 suggests that Ubuntu/Debian has a package shipping said tool.

[samccann]

FWIW there is a flag to turn off breadcrumbs. As I recall, that's the real memory/cpu hog in the package docs build, but pinging @felixfontein because I could be wrong.

He can also pipe in on whether turning off breadcrumbs to simplify the test build is still a valid test of the docs build with relaxed constraints.

[webknjaz]

I'd suggest invoking Sphinx under py-spy to track down what consumes most resources.

[felixfontein]

Turning breadcrumbs off should improve performance and still be relatively valid. Other "optimizations", such as using HTML tables, are not a good idea. (In fact I'm thinking of deprecating that option.)

[webknjaz]

I remembered one of the former MongoDB folks showing their Sphinx optimizations on the Write The Docs Slack instance. So I asked to remind me of that and it was https://github.com/mongodb/docs-tools/blob/2fbc633/sphinxext/fasthtml.py#L33-L149. Not sure if it's applicable to our case or is relevant in the context of modern Sphinx but worth a shot probably. Still, it'd be useful to know where the bottleneck is...

[oraNod]

Going to close this issue because the job is updated to allow using relaxed requirements:

I'll create a new/separate issue for the Sphinx optimization work and link it to this one.