search

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
883 stars 237 forks source link

toil-cwl-runner --maxLogFileSize RuntimeError #4886

Closed RomGarand closed 2 months ago

RomGarand commented 2 months ago

Hello,

I get the following error when I try to run toil-cwl-runner with --maxLogFileSize=infinity :

RuntimeError: infinity not a valid unit, valid units are ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib', 'b', 'k', 'm', 'g', 't', 'p', 'e', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb'].

Previously, Toil told me I could use the term "infinity" for this property :

toil-cwl-runner: error: --maxLogFileSize (0) must be within the range: [1, infinity)

Is it a bug ? Or do I need to put a high value to simulate infinity ?

Here the traceback : 

exec /usr/local/bin/toil-cwl-runner --jobStore=file:/work/ALT/ssalto_dev/chronos-valid-dev/chronos/job_stores/l0a_hr_prime_ti0000/20240429T123440.496004 --logDebug --batchSystem=mesos --mesosMaster=chronos-scheduler:5050 --mesosRole=PRIORITY_127 --clean=never --cleanWorkDir=always --preserve-entire-environment --outdir=/work/ALT/ssalto_dev/chronos-valid-dev/chronos/outdir/l0a_hr_prime_ti0000/20240429T123440.496004 --disableCaching --linkImports --disableHotDeployment --moveExports --maxLogFileSize=infinity --retryCount=10pb /data/configuration/swot-karin-ti0000/cwl/wf_l0a_prime_batch.cwl /work/ALT/ssalto_dev/chronos-valid-dev/chronos/contexts/l0a_hr_prime_ti0000/2024/04/29/20240429T123440.496004/args.json
Traceback (most recent call last):
  File "/usr/local/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 3531, in main
    options = get_options(args)
  File "/usr/local/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 3517, in get_options
    options, cwl_options = parser.parse_known_args(args)
  File "/usr/local/lib/python3.9/site-packages/configargparse.py", line 979, in parse_known_args
    namespace, unknown_args = argparse.ArgumentParser.parse_known_args(
  File "/usr/local/lib/python3.9/argparse.py", line 1851, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/lib/python3.9/argparse.py", line 2060, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/local/lib/python3.9/argparse.py", line 2000, in consume_optional
    take_action(action, args, option_string)
  File "/usr/local/lib/python3.9/argparse.py", line 1912, in take_action
    argument_values = self._get_values(action, argument_strings)
  File "/usr/local/lib/python3.9/argparse.py", line 2443, in _get_values
    value = self._get_value(action, arg_string)
  File "/usr/local/lib/python3.9/argparse.py", line 2476, in _get_value
    result = type_func(arg_string)
  File "/usr/local/lib/python3.9/site-packages/toil/options/common.py", line 531, in <lambda>
    h2b = lambda x: human2bytes(str(x))
  File "/usr/local/lib/python3.9/site-packages/toil/lib/conversions.py", line 76, in human2bytes
    value, unit = parse_memory_string(string)
  File "/usr/local/lib/python3.9/site-packages/toil/lib/conversions.py", line 66, in parse_memory_string
    raise RuntimeError(f"{units} not a valid unit, valid units are {VALID_PREFIXES}.")
RuntimeError: infinity not a valid unit, valid units are ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib', 'b', 'k', 'm', 'g', 't', 'p', 'e', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb'].

Thanks for helping !

Romain

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1547

adamnovak commented 2 months ago

I think this is a bug where the output we produce describing what the option can be is confusing. We only accept actual numbers, possibly with units, but we write out the upper bound as infinity with a ).

If you don't read that as an open, end-exclusive interval because of the ), you might be led to believe that the string "infinity" is allowed, when it isn't.

We should change the message to must be greater than 0, and not mention an upper limit.

We also could consider actually allowing you to set an unlimited --maxLogFileSize, which seems to be what you are trying to accomplish. Until we can do that, you can try --maxLogFileSize 1EiB, because you probably do not have an exabyte of storage.

RomGarand commented 2 months ago

I understand. I will follow your recommendation and try --maxLogFileSize 1EiB . Thank you for your reply and your responsiveness / quick support