Open fmigneault opened 1 year ago
mr-c
commented
1 year ago As a workaround, I suggest giving your enums a name when reusing them. Also consider using SchemaDefRequirement.
hints:
WPS1Requirement:
provider: https://finch.crim.ca/wps
process: ensemble_grid_point_wetdaysPlease use a namespace for any extensions, like WPS1Requirement
fmigneault
commented
1 year ago @mr-c
I have tried setting name for the enum. While this "works", cwltool replaces all the definitions it finds with a simple reference to the tmp schema by name, resulting in something like:
type:
- "null"
- 'tmp.tmp2utae6vd.ensemble_grid_point_wetdays.scenario.scenariobd96176e-aafc-4fff-b32d-261c2bc4d858'
- type: array
items: 'tmp.tmp2utae6vd.ensemble_grid_point_wetdays.scenario.scenariobd96176e-aafc-4fff-b32d-261c2bc4d858'Because of my current implementation, I'm having issues resolving this type (other steps down the line have errors with this "unknown" string).
Now, to my question...
When I run cwltool from the command line with definitions that do NOT have name, it successfully executes the operation without any error (see down below).
What could be at cause that schema fails when using Factory.make, but not when using cwltool CLI?
Following are the adapted tool and inputs to test without our custom definitions:
# /tmp/tool.cwl
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
requirements:
InlineJavascriptRequirement: {}
inputs:
- id: lat
type:
- string
- type: array
items: string
- id: lon
type:
- string
- type: array
items: string
- id: start_date
type:
- 'null'
- string
- id: end_date
type:
- 'null'
- string
- id: ensemble_percentiles
type:
- 'null'
- string
default: 10,50,90
- id: average
type:
- 'null'
- boolean
default: false
- id: dataset
type:
- 'null'
- type: enum
symbols:
- humidex-daily
- candcs-u5
- candcs-u6
- bccaqv2
default: candcs-u5
- id: scenario
type:
- 'null'
- type: enum
symbols:
- ssp126
- rcp85
- rcp45
- rcp26
- ssp585
- ssp245
- type: array
items:
type: enum
symbols:
- ssp126
- rcp85
- rcp45
- rcp26
- ssp585
- ssp245
- id: models
type:
- 'null'
- type: enum
symbols:
- KACE-1-0-G
- CCSM4
- MIROC5
- EC-Earth3-Veg
- TaiESM1
- GFDL-ESM4
- GFDL-CM3
- CanESM5
- HadGEM3-GC31-LL
- INM-CM4-8
- IPSL-CM5A-MR
- EC-Earth3
- GFDL-ESM2G
- humidex_models
- GFDL-ESM2M
- MIROC-ESM
- CSIRO-Mk3-6-0
- MPI-ESM-LR
- NorESM1-M
- CNRM-CM5
- all
- GISS-E2-1-G
- 24models
- MPI-ESM1-2-HR
- CNRM-ESM2-1
- CNRM-CM6-1
- CanESM2
- FGOALS-g3
- NorESM1-ME
- IPSL-CM6A-LR
- CMCC-ESM2
- pcic12
- EC-Earth3-Veg-LR
- ACCESS-ESM1-5
- MRI-CGCM3
- MIROC-ESM-CHEM
- NorESM2-MM
- bcc-csm1-1-m
- BNU-ESM
- UKESM1-0-LL
- CESM1-CAM5
- MIROC-ES2L
- MRI-ESM2-0
- HadGEM2-ES
- MIROC6
- MPI-ESM-MR
- INM-CM5-0
- bcc-csm1-1
- BCC-CSM2-MR
- ACCESS-CM2
- NorESM2-LM
- IPSL-CM5A-LR
- FGOALS-g2
- HadGEM2-AO
- 26models
- MPI-ESM1-2-LR
- KIOST-ESM
- type: array
items:
type: enum
symbols:
- KACE-1-0-G
- CCSM4
- MIROC5
- EC-Earth3-Veg
- TaiESM1
- GFDL-ESM4
- GFDL-CM3
- CanESM5
- HadGEM3-GC31-LL
- INM-CM4-8
- IPSL-CM5A-MR
- EC-Earth3
- GFDL-ESM2G
- humidex_models
- GFDL-ESM2M
- MIROC-ESM
- CSIRO-Mk3-6-0
- MPI-ESM-LR
- NorESM1-M
- CNRM-CM5
- all
- GISS-E2-1-G
- 24models
- MPI-ESM1-2-HR
- CNRM-ESM2-1
- CNRM-CM6-1
- CanESM2
- FGOALS-g3
- NorESM1-ME
- IPSL-CM6A-LR
- CMCC-ESM2
- pcic12
- EC-Earth3-Veg-LR
- ACCESS-ESM1-5
- MRI-CGCM3
- MIROC-ESM-CHEM
- NorESM2-MM
- bcc-csm1-1-m
- BNU-ESM
- UKESM1-0-LL
- CESM1-CAM5
- MIROC-ES2L
- MRI-ESM2-0
- HadGEM2-ES
- MIROC6
- MPI-ESM-MR
- INM-CM5-0
- bcc-csm1-1
- BCC-CSM2-MR
- ACCESS-CM2
- NorESM2-LM
- IPSL-CM5A-LR
- FGOALS-g2
- HadGEM2-AO
- 26models
- MPI-ESM1-2-LR
- KIOST-ESM
default: all
- id: thresh
type:
- 'null'
- string
default: 1.0 mm/day
- id: freq
type:
- 'null'
- type: enum
symbols:
- YS
- QS-DEC
- AS-JUL
- MS
default: YS
- id: op
type:
- 'null'
- type: enum
symbols:
- '>='
- '>'
- gt
- ge
default: '>='
- id: month
type:
- 'null'
- int
- type: array
items: int
inputBinding:
valueFrom: "\n ${\n const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12];\n if (Array.isArray(self)) {\n \n if (self.every(item => values.includes(item))) {\n return self;\n }\n else {\n throw \"invalid value(s) in [\" + self + \"] are not all allowed values from [\" + values + \"]\";\n }\n \n }\n else {\n \n if (values.includes(self)) {\n return self;\n }\n else {\n throw \"invalid value \" + self + \" is not an allowed value from [\" + values + \"]\";\n }\n \n }\n }\n "
- id: season
type:
- 'null'
- type: enum
symbols:
- SON
- MAM
- JJA
- DJF
- id: check_missing
type:
- 'null'
- type: enum
symbols:
- pct
- at_least_n
- wmo
- skip
- from_context
- any
default: any
- id: missing_options
type:
- 'null'
- File
format: iana:application/json
- id: cf_compliance
type:
- 'null'
- type: enum
symbols:
- raise
- log
- warn
default: warn
- id: data_validation
type:
- 'null'
- type: enum
symbols:
- raise
- log
- warn
default: raise
- id: output_name
type:
- 'null'
- string
- id: output_format
type:
- 'null'
- type: enum
symbols:
- csv
- netcdf
default: netcdf
- id: csv_precision
type:
- 'null'
- int
outputs:
- id: output
type: stdout
$namespaces:
iana: https://www.iana.org/assignments/media-types/
edam: http://edamontology.org/
# /tmp/data.yml
lat: "45.35629610945964"
lon: "-73.98748912005094"
average: "False"
start_date: "1950"
end_date: "1960"
ensemble_percentiles: ""
dataset: "candcs-u6"
scenario: "ssp126"
models: "26models"
freq: "YS"
data_validation: "warn"
output_format: "csv"
csv_precision: 0
thresh: "15 mm\/day"cwltool /tmp/tool.cwl /tmp/data.yml
INFO /home/francis/dev/conda/envs/weaver/bin/cwltool 3.1.20230906142556
INFO Resolved '/tmp/tool.cwl' to 'file:///tmp/tool.cwl'
WARNING ../../tmp/tool.cwl:222:5: JSHINT: const values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12];
../../tmp/tool.cwl:222:5: JSHINT: ^
../../tmp/tool.cwl:222:5: JSHINT: W104: 'const' is available in ES. CWL only supports ES5.1
WARNING ../../tmp/tool.cwl:222:5: JSHINT: if (self.every(item => values.includes(item))) {
../../tmp/tool.cwl:222:5: JSHINT: ^
../../tmp/tool.cwl:222:5: JSHINT: W119: 'arrow function syntax (=>)' is only available in ES6. CWL only supports ES5.1
INFO [job tool.cwl] /tmp/ctjoy0z1$ echo > /tmp/ctjoy0z1/263e318e7095de94ae5eaecd5f725701aee65c2d
INFO [job tool.cwl] completed success
{
"output": {
"location": "file:///home/francis/263e318e7095de94ae5eaecd5f725701aee65c2d",
"basename": "263e318e7095de94ae5eaecd5f725701aee65c2d",
"class": "File",
"checksum": "sha1$adc83b19e793491b1c6ea0fd8b46cd9f32e592fc",
"size": 1,
"path": "/home/francis/263e318e7095de94ae5eaecd5f725701aee65c2d"
}
}INFO Final process status is success
If possible, I would like to replicate this behaviour where the type: ["null", <enum>, enum[] ] combined succeeds validation as above.
Implementing proper SchemaDefRequirement would be a follow-up step.
Is there some option to set to achieve this goal?
Thanks
Description
I have a CWL definition as presented below that reuses
type: enumin multiple definitions to allow null (optional), single-value and an array of those values for some of my inputs.Expected Behavior
The CWL definition should be valid.
Actual Behavior
For some reason, schema salad raises an error indicating that
'name': 'scenario1b4d8bed-8ce2-4a0b-a0ad-194203993920'(schema auto-generated fromscenarioinput definition?) already exists. I wonder if this is a false-positive because of the definitions repeats the exact sameenumportion for the single value vs multi-value array types?Workflow Code
Full Traceback
Your Environment
cwltool --version