common-workflow-language / user_guide

The CWL v1.0 - v1.2 user guide
http://www.commonwl.org/user_guide/
Other
42 stars 67 forks source link

Automate generation of outputs #248

Closed kinow closed 2 years ago

kinow commented 2 years ago

Closes #148

kinow commented 2 years ago

This PR uses a patched version of https://github.com/sphinx-contrib/sphinxcontrib-runcmd. The patches include:

The directive is not actively maintained, with low code activity. I will prepare pull requests over the next weeks/months, but the author may decide not to accept every PR (also, the directive itself is a wrapper around code-block, so we could even build another directive on top of it, instead of patching it, when/if we have more time).

Here's an example of what we had before:

```{code-block} console
:name: running-hello_world.cwl-with-cwltool
:caption: Running `hello_world.cwl` with `cwltool`.

$ cwltool hello_world.cwl
INFO /tmp/venv/bin/cwltool 3.1.20220628170238
INFO Resolved 'hello_world.cwl' to 'file:///tmp/hello_world.cwl'
INFO [job hello_world.cwl] /tmp/yn0e8xu6$ echo \
    'Hello World'
Hello World
INFO [job hello_world.cwl] completed success
{}
INFO Final process status is success

Which renders to:

![image](https://user-images.githubusercontent.com/304786/188783851-3fb7e2a3-dbac-4675-9391-b8b3543506ad.png)

With the new directive, we have:
:name: running-hello_world.cwl-with-cwltool
:caption: Running `hello_world.cwl` with `cwltool`.


Which renders to:

![image](https://user-images.githubusercontent.com/304786/188784061-d43a9d02-e551-4a73-9f95-5166bc22718d.png)
kinow commented 2 years ago

As for the second part, I forgot we are building the final version on ReadTheDocs, which already uses Docker :smile:

Right now it's not able to locate the included files, so I believe its default working directory is different than what I am using. Will play with that value, but quite close to having a working preview before updating the other files.

kinow commented 2 years ago

Preview of the same part of the site, but now rendered in ReadTheDocs (note that there's no /tmp or /home/kinow):

image

We have full control of the output, so in theory we could even find-replace that value for something else if needed, but that makes the build a little more brittle I guess.

Any thoughts so far @mr-c, @tetron?

mr-c commented 2 years ago

This is great to see! A big bonus if the explicit color coding can be maintained instead of random highlighting by the parser.

kinow commented 2 years ago

This is great to see! A big bonus if the explicit color coding can be maintained instead of random highlighting by the parser.

Good idea. I know ~colorama~ ansi2html is able to convert the shell output into HTML, let me see if I can put everything together and get the same terminal colors into the browser… :crossed_fingers:

kinow commented 2 years ago

I quickly hacked the Sphinx directive to import and call ansi2html, before trying to integrate its output into Sphinx rendered output. The ansi2html output doesn't look very promising.

image

Going to push this back to the backburner, for a follow-up :+1: :disappointed_relieved:

kinow commented 2 years ago

50% of the pages converted to use {runcmd}. Going to finish it between tomorrow and Monday, then review again the generated pages, and finally confirm the conformance-tests.yml is not missing anything. Then it will be ready for review.

kinow commented 2 years ago

Ready for review!

Found a few typos here and there, and some examples that were broken. Also found some code blocks that were not using the caption and instead had extra HTML or a paragraph with the name of the file. There were also a few backtick code blocks (i.e. ``) instead of using the directive{code-block}`.

Also updated the names of the directories in the src/_includes/cwl folder, replacing the old 05-custom-types by the same name of the markdown file, custom-types. In the markdown file we have :working-directory: src/_includes/cwl/custom-types so that the command doesn't need to include the whole path, but given we have a convention now of directories matching the markdown names, we could even automate that too, by trying to change the working directory if a directory like src/_includes/cwl/${FILE_NAME} exists — for some day :+1:

The output of the commands now must be dynamically generated during build by ReadTheDocs :+1:

kinow commented 2 years ago

@mr-c removed the TODO tags, replaced cwl-runner by cwltool, and used meld to diff the _build/html directories generated with make html on this branch and on main. It looks ready for review… except…

The ReadTheDocs pages are broken due to filelock module? I guess that's something new?

image

I was going to add it to the user guide project as a dependency, but I think maybe a better suggestion would be to fix that in cwltool, if that's indeed a dependency that was added recently but not included in the setup files?

kinow commented 2 years ago

It looks like CacheControl is used in cwltool transitively, and they recently switched from lockfile to filelock :thinking: https://github.com/ionrock/cachecontrol/commit/f0cd6b49e5c4215137e5c24ac0b46752f0c61e69

I destroyed my venv and re-created it, then installed with pip install -e ., and looking at pip list and pipdeptree (after installing it manually), I can confirm filelock is missing and I get the same error as in RTD. Maybe some recent change in cwltool and/or schema-salad broke it? Output of pipdeptree below.

(venv) kinow@ranma:~/Development/python/workspace/cwltool$ pipdeptree 
cwltool==3.1.20220907202945
  - argcomplete [required: Any, installed: 2.0.0]
  - bagit [required: >=1.6.4, installed: 1.8.1]
  - coloredlogs [required: Any, installed: 15.0.1]
    - humanfriendly [required: >=9.1, installed: 10.0]
  - cwl-utils [required: >=0.15, installed: 0.18]
    - CacheControl [required: Any, installed: 0.12.12]
      - msgpack [required: >=0.5.2, installed: 1.0.4]
      - requests [required: Any, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
        - charset-normalizer [required: >=2,<3, installed: 2.1.1]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
    - cwl-upgrader [required: >=1.2.3, installed: 1.2.4]
      - ruamel.yaml [required: >=0.15.78,<0.17.22, installed: 0.17.21]
        - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
      - ruamel.yaml [required: >=0.15.98,<0.17.22, installed: 0.17.21]
        - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
      - ruamel.yaml [required: >=0.15.71,<0.17.22, installed: 0.17.21]
        - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
      - schema-salad [required: Any, installed: 8.3.20220909144501]
        - CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
          - msgpack [required: >=0.5.2, installed: 1.0.4]
          - requests [required: Any, installed: 2.28.1]
            - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
            - charset-normalizer [required: >=2,<3, installed: 2.1.1]
            - idna [required: >=2.5,<4, installed: 3.3]
            - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
        - lockfile [required: >=0.9, installed: 0.12.2]
        - mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
        - rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
          - isodate [required: Any, installed: 0.6.1]
            - six [required: Any, installed: 1.16.0]
          - pyparsing [required: Any, installed: 3.0.9]
          - setuptools [required: Any, installed: 58.1.0]
        - requests [required: >=1.0, installed: 2.28.1]
          - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
          - charset-normalizer [required: >=2,<3, installed: 2.1.1]
          - idna [required: >=2.5,<4, installed: 3.3]
          - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
        - ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
          - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
        - ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
          - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
      - setuptools [required: Any, installed: 58.1.0]
    - packaging [required: Any, installed: 21.3]
      - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
    - rdflib [required: Any, installed: 6.2.0]
      - isodate [required: Any, installed: 0.6.1]
        - six [required: Any, installed: 1.16.0]
      - pyparsing [required: Any, installed: 3.0.9]
      - setuptools [required: Any, installed: 58.1.0]
    - requests [required: Any, installed: 2.28.1]
      - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
      - charset-normalizer [required: >=2,<3, installed: 2.1.1]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
    - schema-salad [required: >=8.3.20220825114525,<9, installed: 8.3.20220909144501]
      - CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
        - msgpack [required: >=0.5.2, installed: 1.0.4]
        - requests [required: Any, installed: 2.28.1]
          - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
          - charset-normalizer [required: >=2,<3, installed: 2.1.1]
          - idna [required: >=2.5,<4, installed: 3.3]
          - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
      - lockfile [required: >=0.9, installed: 0.12.2]
      - mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
      - rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
        - isodate [required: Any, installed: 0.6.1]
          - six [required: Any, installed: 1.16.0]
        - pyparsing [required: Any, installed: 3.0.9]
        - setuptools [required: Any, installed: 58.1.0]
      - requests [required: >=1.0, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
        - charset-normalizer [required: >=2,<3, installed: 2.1.1]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
      - ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
        - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
      - ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
        - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
  - mypy-extensions [required: Any, installed: 0.4.3]
  - prov [required: ==1.5.1, installed: 1.5.1]
    - lxml [required: Any, installed: 4.9.1]
    - networkx [required: Any, installed: 2.8.6]
    - python-dateutil [required: Any, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - rdflib [required: >=4.2.1, installed: 6.2.0]
      - isodate [required: Any, installed: 0.6.1]
        - six [required: Any, installed: 1.16.0]
      - pyparsing [required: Any, installed: 3.0.9]
      - setuptools [required: Any, installed: 58.1.0]
    - six [required: >=1.9.0, installed: 1.16.0]
  - psutil [required: >=5.6.6, installed: 5.9.2]
  - pydot [required: >=1.4.1, installed: 1.4.2]
    - pyparsing [required: >=2.1.4, installed: 3.0.9]
  - pyparsing [required: !=3.0.2, installed: 3.0.9]
  - rdflib [required: >=4.2.2,<6.3.0, installed: 6.2.0]
    - isodate [required: Any, installed: 0.6.1]
      - six [required: Any, installed: 1.16.0]
    - pyparsing [required: Any, installed: 3.0.9]
    - setuptools [required: Any, installed: 58.1.0]
  - requests [required: >=2.6.1, installed: 2.28.1]
    - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
    - charset-normalizer [required: >=2,<3, installed: 2.1.1]
    - idna [required: >=2.5,<4, installed: 3.3]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
  - ruamel.yaml [required: >=0.15,<0.17.22, installed: 0.17.21]
    - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
  - schema-salad [required: >=8.2.20211104054942,<9, installed: 8.3.20220909144501]
    - CacheControl [required: >=0.11.7,<0.13, installed: 0.12.12]
      - msgpack [required: >=0.5.2, installed: 1.0.4]
      - requests [required: Any, installed: 2.28.1]
        - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
        - charset-normalizer [required: >=2,<3, installed: 2.1.1]
        - idna [required: >=2.5,<4, installed: 3.3]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
    - lockfile [required: >=0.9, installed: 0.12.2]
    - mistune [required: >=0.8.1,<0.9, installed: 0.8.4]
    - rdflib [required: >=4.2.2,<7.0.0, installed: 6.2.0]
      - isodate [required: Any, installed: 0.6.1]
        - six [required: Any, installed: 1.16.0]
      - pyparsing [required: Any, installed: 3.0.9]
      - setuptools [required: Any, installed: 58.1.0]
    - requests [required: >=1.0, installed: 2.28.1]
      - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
      - charset-normalizer [required: >=2,<3, installed: 2.1.1]
      - idna [required: >=2.5,<4, installed: 3.3]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
    - ruamel.yaml [required: >=0.17.6,<0.17.22, installed: 0.17.21]
      - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
    - ruamel.yaml [required: >=0.16.12,<0.17.22, installed: 0.17.21]
      - ruamel.yaml.clib [required: >=0.2.6, installed: 0.2.6]
  - setuptools [required: Any, installed: 58.1.0]
  - shellescape [required: >=3.4.1,<3.9, installed: 3.8.1]
  - typing-extensions [required: Any, installed: 4.3.0]
pip==22.0.4
pipdeptree==2.3.1
kinow commented 2 years ago

But other than that this PR should be ready to review :grimacing:

mr-c commented 2 years ago

The broken cachecontrol package has been yanked. Additionally I released a new version of schema-salad that will avoid this problem in the future

mr-c commented 2 years ago

ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information. The error was: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim

From https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/expressions.html

kinow commented 2 years ago

ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information. The error was: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim

From https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/expressions.html

Great catch!!! Fixed by installing nodejs on RTD, thanks!!!

image

kinow commented 2 years ago

The broken cachecontrol package has been yanked. Additionally I released a new version of schema-salad that will avoid this problem in the future

Thanks for that @mr-c, the build is now fixed on RTD. I've fixed your other feedback. It should be ready for review again :D :drum: :crossed_fingers:

mr-c commented 2 years ago
$ cwltool arguments.cwl arguments-job.yml
INFO /home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/envs/248/bin/cwltool 3.1.20220913185150
INFO Resolved 'arguments.cwl' to 'file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/248/src/_includes/cwl/additional-arguments-and-parameters/arguments.cwl'
ERROR Workflow error, try again with --debug for more information:
Docker is not available for this tool, try --no-container to disable Docker, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: docker executable is not available

https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/additional-arguments-and-parameters.html

kinow commented 2 years ago
$ cwltool arguments.cwl arguments-job.yml
INFO /home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/envs/248/bin/cwltool 3.1.20220913185150
INFO Resolved 'arguments.cwl' to 'file:///home/docs/checkouts/readthedocs.org/user_builds/common-workflow-languageuser-guide/checkouts/248/src/_includes/cwl/additional-arguments-and-parameters/arguments.cwl'
ERROR Workflow error, try again with --debug for more information:
Docker is not available for this tool, try --no-container to disable Docker, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: docker executable is not available

https://common-workflow-languageuser-guide--248.org.readthedocs.build/en/248/topics/additional-arguments-and-parameters.html

This one was trickier, especially since we are running RTD builds already inside Docker, and even though docker-in-docker works, I would prefer to avoid that.

The best solution I could think of was udocker, @mr-c. WDYT? I modified the script to re-use the existing CWLTOOL_OPTIONS, adding just the option to remove colors. Then, in RTD settings I set the environment variable to use udocker.

This way, anyone building the User Guide can use udocker, podman, docker, etc. But on RTD, it will always use udocker.

Screenshot from 2022-09-14 12-45-33 Screenshot from 2022-09-14 12-38-53

mr-c commented 2 years ago

Does it use Docker properly when rendered with GitHub Pages?

kinow commented 2 years ago

Does it use Docker properly when rendered with GitHub Pages?

Ah, another good catch @mr-c. I've added the environment variable for GH Pages too, since it'll be using a Docker container for building the docs.

I've pushed the commit here, but then added an extra commit and pushed only to my fork. Then enabled the workflow to run on my fork, so that we could have a preview before merging: https://kinow.github.io/user_guide/topics/additional-arguments-and-parameters.html

https://github.com/kinow/user_guide/actions (see commit with message "f" - for some unknown reason all my temporary commits have this "f" message :man_shrugging: :smile: ).

Thanks!

mr-c commented 2 years ago

Thank you @kinow !

kinow commented 2 years ago

Thank you @kinow !

Thank you for reviewing and testing the containers on GH. I liked that GH action improvement to prepull the containers!

mr-c commented 2 years ago

I liked that GH action improvement to prepull the containers!

👍 It was necessary to keep the docker pull output out of the cwltool output