Calling cwltool.main() with preparsed args and provenance fails unless unparsed args are also provided

[davidjsherman]

Expected Behavior

Calling cwltool.main() with preparsed args only, but including --provenance, should execute and create a RO-Crate, the same as when running as a script.

Actual Behavior

The execution fails, raising Exception("argsl cannot be None")

Workflow Code

Program prov-no-argsl.py:

import sys

from cwltool.main import main as cwlmain
from cwltool.argparser import arg_parser

unparsed_args = ["--provenance", "ro-crate", "hello_world.cwl", "--message=Hello"]
parsed_args = arg_parser().parse_args(unparsed_args)

cwlmain(args=parsed_args)

Assuming hello_world.cwl is in the working directory

cwlVersion: v1.2
class: CommandLineTool
baseCommand: echo
inputs:
  message:
    type: string
    default: Hello World
    inputBinding:
      position: 1
outputs: []

Full Traceback

INFO prov-no-argsl.py 3.1.20230719185429
Traceback (most recent call last):
  File "/private/tmp/cwl-prov/prov-no-argsl.py", line 9, in <module>
    cwlmain(args=parsed_args)
  File "/opt/homebrew/lib/python3.11/site-packages/cwltool/main.py", line 1052, in main
    raise Exception("argsl cannot be None")
Exception: argsl cannot be None

Your Environment

• cwltool version: /opt/homebrew/bin/cwltool 3.1.20230719185429

Remarks

Note that if the unparsed arguments are also provided, then execution proceeds as expected

Program prov-with-argsl.py:

import sys

from cwltool.argparser import arg_parser
from cwltool.main import main as cwlmain

unparsed_args = ["--provenance", "ro-crate", "hello_world.cwl", "--message=Hello"]
parsed_args = arg_parser().parse_args(unparsed_args)

cwlmain(args=parsed_args, argsl=unparsed_args)

INFO prov-with-argsl.py 3.1.20230719185429
INFO [cwltool] prov-with-argsl.py --provenance ro-crate hello_world.cwl --message=Hello
INFO Resolved 'hello_world.cwl' to 'file:///private/tmp/cwl-prov/hello_world.cwl'
INFO [job hello_world.cwl] /private/tmp/docker_tmpg0yk4mky$ echo \
    Hello
Hello
INFO [job hello_world.cwl] completed success
/opt/homebrew/lib/python3.11/site-packages/rdflib/plugins/serializers/nt.py:40: UserWarning: NTSerializer always uses UTF-8 encoding. Given encoding was: None
  warnings.warn(
{}INFO Final process status is success
INFO [provenance] Finalizing Research Object
INFO [provenance] Deleting existing /private/tmp/cwl-prov/ro-crate
INFO [provenance] Research Object saved to /private/tmp/cwl-prov/ro-crate

Note further that it is only when requesting provenance that this error occurs.

Program no-prov.py:

import sys

from cwltool.main import main as cwlmain
from cwltool.argparser import arg_parser

unparsed_args = ["hello_world.cwl", "--message=Hello"]
parsed_args = arg_parser().parse_args(unparsed_args)

cwlmain(args=parsed_args)

INFO no-prov.py 3.1.20230719185429
INFO Resolved 'hello_world.cwl' to 'file:///private/tmp/cwl-prov/hello_world.cwl'
INFO [job hello_world.cwl] /private/tmp/docker_tmpt2b9sgm2$ echo \
    Hello
Hello
INFO [job hello_world.cwl] completed success
{}INFO Final process status is success

[davidjsherman]

This blocks requesting provenance when using Calrissian, which calls cwltool.main directly with preparsed arguments and no unparsed arguments.

[cwl-bot]

This issue has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/cwltool-main-text-arguments-a-hard-requirement-for-provenance/862/2

[mr-c]

Dear @davidjsherman , thank you for the issue and your interest in supporting CWLProv on Calrissian!

I think this is a typing error and I would accept a PR to fix it by removing the exception and adjusting the types to be Optional.

[cwl-bot]

This issue has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/cwltool-main-text-arguments-a-hard-requirement-for-provenance/862/3