Closed kinow closed 2 years ago
This PR uses a patched version of https://github.com/sphinx-contrib/sphinxcontrib-runcmd. The patches include:
$
to the command executed, similarly to what we have with the {code-block}
directive;cd src/_includes/cwl/
before running the specified command);syntax: bash
instead of text (since we will use it for running commands, that makes more sense);The directive is not actively maintained, with low code activity. I will prepare pull requests over the next weeks/months, but the author may decide not to accept every PR (also, the directive itself is a wrapper around code-block
, so we could even build another directive on top of it, instead of patching it, when/if we have more time).
Here's an example of what we had before:
```{code-block} console
:name: running-hello_world.cwl-with-cwltool
:caption: Running `hello_world.cwl` with `cwltool`.
$ cwltool hello_world.cwl
INFO /tmp/venv/bin/cwltool 3.1.20220628170238
INFO Resolved 'hello_world.cwl' to 'file:///tmp/hello_world.cwl'
INFO [job hello_world.cwl] /tmp/yn0e8xu6$ echo \
'Hello World'
Hello World
INFO [job hello_world.cwl] completed success
{}
INFO Final process status is success
Which renders to:
![image](https://user-images.githubusercontent.com/304786/188783851-3fb7e2a3-dbac-4675-9391-b8b3543506ad.png)
With the new directive, we have:
:name: running-hello_world.cwl-with-cwltool
:caption: Running `hello_world.cwl` with `cwltool`.
Which renders to:
![image](https://user-images.githubusercontent.com/304786/188784061-d43a9d02-e551-4a73-9f95-5166bc22718d.png)
As for the second part, I forgot we are building the final version on ReadTheDocs, which already uses Docker :smile:
Right now it's not able to locate the included files, so I believe its default working directory is different than what I am using. Will play with that value, but quite close to having a working preview before updating the other files.
Preview of the same part of the site, but now rendered in ReadTheDocs (note that there's no /tmp
or /home/kinow
):
We have full control of the output, so in theory we could even find-replace that value for something else if needed, but that makes the build a little more brittle I guess.
Any thoughts so far @mr-c, @tetron?
This is great to see! A big bonus if the explicit color coding can be maintained instead of random highlighting by the parser.
This is great to see! A big bonus if the explicit color coding can be maintained instead of random highlighting by the parser.
Good idea. I know ~colorama~ ansi2html is able to convert the shell output into HTML, let me see if I can put everything together and get the same terminal colors into the browser… :crossed_fingers:
I quickly hacked the Sphinx directive to import and call ansi2html, before trying to integrate its output into Sphinx rendered output. The ansi2html
output doesn't look very promising.
Going to push this back to the backburner, for a follow-up :+1: :disappointed_relieved:
50% of the pages converted to use {runcmd}
. Going to finish it between tomorrow and Monday, then review again the generated pages, and finally confirm the conformance-tests.yml
is not missing anything. Then it will be ready for review.
Ready for review!
Found a few typos here and there, and some examples that were broken. Also found some code blocks that were not using the caption
and instead had extra HTML or a paragraph with the name of the file. There were also a few backtick code blocks (i.e. ``) instead of using the directive
{code-block}`.
Also updated the names of the directories in the src/_includes/cwl
folder, replacing the old 05-custom-types
by the same name of the markdown file, custom-types
. In the markdown file we have :working-directory: src/_includes/cwl/custom-types
so that the command doesn't need to include the whole path, but given we have a convention now of directories matching the markdown names, we could even automate that too, by trying to change the working directory if a directory like src/_includes/cwl/${FILE_NAME}
exists — for some day :+1:
The output of the commands now must be dynamically generated during build by ReadTheDocs :+1:
@mr-c removed the TODO
tags, replaced cwl-runner
by cwltool
, and used meld
to diff the _build/html
directories generated with make html
on this branch and on main
. It looks ready for review… except…
The ReadTheDocs pages are broken due to filelock
module? I guess that's something new?
I was going to add it to the user guide project as a dependency, but I think maybe a better suggestion would be to fix that in cwltool
, if that's indeed a dependency that was added recently but not included in the setup files?
It looks like CacheControl
is used in cwltool
transitively, and they recently switched from lockfile to filelock :thinking: https://github.com/ionrock/cachecontrol/commit/f0cd6b49e5c4215137e5c24ac0b46752f0c61e69
I destroyed my venv
and re-created it, then installed with pip install -e .
, and looking at pip list
and pipdeptree
(after installing it manually), I can confirm filelock
is missing and I get the same error as in RTD. Maybe some recent change in cwltool and/or schema-salad broke it? Output of pipdeptree
below.
(venv) kinow@ranma:~/Development/python/workspace/cwltool$ pipdeptree
cwltool==3.1.20220907202945
- argcomplete [required: Any, installed: 2.0.0]
- bagit [required: >=1.6.4, installed: 1.8.1]
- coloredlogs [required: Any, installed: 15.0.1]
- humanfriendly [required: >=9.1, installed: 10.0]
- cwl-utils [required: >=0.15, installed: 0.18]
- CacheControl [required: Any, installed: 0.12.12]
- msgpack [required: >=0.5.2, installed: 1.0.4]
- requests [required: Any, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- cwl-upgrader [required: >=1.2.3, installed: 1.2.4]
- ruamel.yaml [required: >=0.15.78,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- ruamel.yaml [required: >=0.15.98,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- ruamel.yaml [required: >=0.15.71,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- schema-salad [required: Any, installed: 8.3.20220909144501]
- CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
- msgpack [required: >=0.5.2, installed: 1.0.4]
- requests [required: Any, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- lockfile [required: >=0.9, installed: 0.12.2]
- mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
- rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- requests [required: >=1.0, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- setuptools [required: Any, installed: 58.1.0]
- packaging [required: Any, installed: 21.3]
- pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
- rdflib [required: Any, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- requests [required: Any, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- schema-salad [required: >=8.3.20220825114525,<9, installed: 8.3.20220909144501]
- CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
- msgpack [required: >=0.5.2, installed: 1.0.4]
- requests [required: Any, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- lockfile [required: >=0.9, installed: 0.12.2]
- mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
- rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- requests [required: >=1.0, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- mypy-extensions [required: Any, installed: 0.4.3]
- prov [required: ==1.5.1, installed: 1.5.1]
- lxml [required: Any, installed: 4.9.1]
- networkx [required: Any, installed: 2.8.6]
- python-dateutil [required: Any, installed: 2.8.2]
- six [required: >=1.5, installed: 1.16.0]
- rdflib [required: >=4.2.1, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- six [required: >=1.9.0, installed: 1.16.0]
- psutil [required: >=5.6.6, installed: 5.9.2]
- pydot [required: >=1.4.1, installed: 1.4.2]
- pyparsing [required: >=2.1.4, installed: 3.0.9]
- pyparsing [required: !=3.0.2, installed: 3.0.9]
- rdflib [required: >=4.2.2,<6.3.0, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- requests [required: >=2.6.1, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- ruamel.yaml [required: >=0.15,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- schema-salad [required: >=8.2.20211104054942,<9, installed: 8.3.20220909144501]
- CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
- msgpack [required: >=0.5.2, installed: 1.0.4]
- requests [required: Any, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- lockfile [required: >=0.9, installed: 0.12.2]
- mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
- rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
- isodate [required: Any, installed: 0.6.1]
- six [required: Any, installed: 1.16.0]
- pyparsing [required: Any, installed: 3.0.9]
- setuptools [required: Any, installed: 58.1.0]
- requests [required: >=1.0, installed: 2.28.1]
- certifi [required: >=2017.4.17, installed: 2022.6.15.1]
- charset-normalizer [required: >=2,<3, installed: 2.1.1]
- idna [required: >=2.5,<4, installed: 3.3]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
- ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
- ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
- setuptools [required: Any, installed: 58.1.0]
- shellescape [required: >=3.4.1,<3.9, installed: 3.8.1]
- typing-extensions [required: Any, installed: 4.3.0]
pip==22.0.4
pipdeptree==2.3.1
But other than that this PR should be ready to review :grimacing:
The broken cachecontrol package has been yanked. Additionally I released a new version of schema-salad that will avoid this problem in the future
ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information. The error was: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim
From https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/expressions.html
ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information. The error was: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim
From https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/expressions.html
Great catch!!! Fixed by installing nodejs
on RTD, thanks!!!
The broken cachecontrol package has been yanked. Additionally I released a new version of schema-salad that will avoid this problem in the future
Thanks for that @mr-c, the build is now fixed on RTD. I've fixed your other feedback. It should be ready for review again :D :drum: :crossed_fingers:
$ cwltool arguments.cwl arguments-job.yml
INFO /home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/envs/248/bin/cwltool 3.1.20220913185150
INFO Resolved 'arguments.cwl' to 'file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/248/src/_includes/cwl/additional-arguments-and-parameters/arguments.cwl'
ERROR Workflow error, try again with --debug for more information:
Docker is not available for this tool, try --no-container to disable Docker, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: docker executable is not available
$ cwltool arguments.cwl arguments-job.yml INFO /home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/envs/248/bin/cwltool 3.1.20220913185150 INFO Resolved 'arguments.cwl' to 'file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/248/src/_includes/cwl/additional-arguments-and-parameters/arguments.cwl' ERROR Workflow error, try again with --debug for more information: Docker is not available for this tool, try --no-container to disable Docker, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: docker executable is not available
This one was trickier, especially since we are running RTD builds already inside Docker, and even though docker-in-docker works, I would prefer to avoid that.
The best solution I could think of was udocker, @mr-c. WDYT? I modified the script to re-use the existing CWLTOOL_OPTIONS, adding just the option to remove colors. Then, in RTD settings I set the environment variable to use udocker.
This way, anyone building the User Guide can use udocker, podman, docker, etc. But on RTD, it will always use udocker.
Does it use Docker properly when rendered with GitHub Pages?
Does it use Docker properly when rendered with GitHub Pages?
Ah, another good catch @mr-c. I've added the environment variable for GH Pages too, since it'll be using a Docker container for building the docs.
I've pushed the commit here, but then added an extra commit and pushed only to my fork. Then enabled the workflow to run on my fork, so that we could have a preview before merging: https://kinow.github.io/user_guide/topics/additional-arguments-and-parameters.html
https://github.com/kinow/user_guide/actions (see commit with message "f" - for some unknown reason all my temporary commits have this "f" message :man_shrugging: :smile: ).
Thanks!
Thank you @kinow !
Thank you @kinow !
Thank you for reviewing and testing the containers on GH. I liked that GH action improvement to prepull the containers!
I liked that GH action improvement to prepull the containers!
👍 It was necessary to keep the docker pull output out of the cwltool output
Closes #148
{code-block} console
that usescwltool
or another command that we would like to run during build (have updated a few files from the beginning of the docs, following the Next links… I think up to the second section of Topics).