Closed alexiswl closed 1 year ago
Solution - find all missing workflows in development_workflows / production_workflows and add them to project.yaml
ica-context-switcher --scope admin --project-name development_workflows
on_ica="$( \
ica workflows list \
--max-items=0 \
--output-format json | \
jq --raw-output \
'
.items |
map(.id) |
sort |
.[]
' \
)"
in_config_yaml="$( \
yq \
'
.projects |
map(
select(
.project_name=="development_workflows"
)
) |
.[] |
(.workflows + .tools) |
map(
.ica_workflow_id
) |
sort |
.[]
' < config/project.yaml \
)"
Use the comm command to see which ones are on ica are not in config yaml
comm -23 \
<(echo $on_ica | tr ' ' '\n') \
<(echo $in_config_yaml | tr ' ' '\n') | \
xargs -I{} \
bash -c "ica workflows get {} -o json | jq -r '.name'"
Yields
bclconvert_dev-wf
custom-filter-vcf_dev-wf
bclconvert-with-qc-pipeline_dev-wf
dragen-somatic-with-germline-pipeline_dev-wf
map_resource_requirements_dev-wf
bclconvert-scatter_dev-wf
bcl-convert_dev-wf
validate-bclconvert-samplesheet_dev-wf
The following are still in PRs so it makes sense they're not in the config yaml on the main branch
bclconvert_dev-wf
bclconvert-with-qc-pipeline_dev-wf
dragen-somatic-with-germline-pipeline_dev-wf
bclconvert-scatter_dev-wf
bcl-convert_dev-wf
validate-bclconvert-samplesheet_dev-wf
map-resource_requirements_dev-wf is present on test_resource_mapping branch from commit c63ff6bf6f30c1b57b51e5791a0e9fccd5f0a27b.
So just custom-filter-vcf_dev-wf
is not present for development workflows.
For production workflows, running the same commands (swapping out development_workflows for production workflows in the project name and context switcher commands), we get the following values for on_ica and in_config_yaml
# on_ica var
wfl.230846758ccf42e3831283ab0e45af0a
wfl.23f61cb1baab412a8c37dc93bed6c2af
wfl.438287b9982744c388aa9ef0136dc59a
wfl.576020a89adb49c3b2081a620d19104d
wfl.714e9172f3674023b210ccc7c47db05a
wfl.7e5ba7470b5549a6b4bf6d95daaa1214
wfl.7ed9c6014ac9498fbcbd4c17c28bc0d4
wfl.87e07ae6b46645a181e04813de535216
wfl.925c7ba0199a4ad6bb5341a3cffd191d
wfl.9e5bdd810ef8404fabb29f8d71214131
wfl.aa0ccece4e004839aa7374d1d6530633
wfl.b41302dd537d44fd92ae719a367d69fd
wfl.f257ca35ced94e648fdda1173144c476
# in config yaml var
wfl.230846758ccf42e3831283ab0e45af0a
wfl.23f61cb1baab412a8c37dc93bed6c2af
wfl.438287b9982744c388aa9ef0136dc59a
wfl.576020a89adb49c3b2081a620d19104d
wfl.714e9172f3674023b210ccc7c47db05a
wfl.7e5ba7470b5549a6b4bf6d95daaa1214
wfl.7ed9c6014ac9498fbcbd4c17c28bc0d4
wfl.87e07ae6b46645a181e04813de535216
wfl.925c7ba0199a4ad6bb5341a3cffd191d
wfl.9e5bdd810ef8404fabb29f8d71214131
wfl.aa0ccece4e004839aa7374d1d6530633
wfl.b41302dd537d44fd92ae719a367d69fd
wfl.f257ca35ced94e648fdda1173144c476
Which are exactly the same, kudos!
@skanwal
Running
ica workflows versions list wfl.a47748cad84d4ca0a41d39a1d40eeb8d
I get
ID NAME LANGUAGE STATUS TIMECREATED
wfv.ab72fd79ee79443b9647cc2db285d929 0.1.0 CWL Draft 2022-10-05 10:45:14.072 +1100 AEDT
Then running
ica workflows versions get wfl.a47748cad84d4ca0a41d39a1d40eeb8d 0.1.0 \
--output-format json | \
jq --raw-output \
'
.definition |
fromjson' | \
yq --prettyPrint --unwrapScalar
I get
class: CommandLineTool
id: '#main'
label: map_resource_requirements v(0.1.0)
doc: |
Documentation for map_resource_requirements v0.1.0
hints:
- dockerPull: bash:5
class: DockerRequirement
- coresMin: 2
ramMin: 4000
class: ResourceRequirement
http://platform.illumina.com/rdf/ica/resources:
type: $(calcInstanceType(inputs.input_size))
size: small
requirements:
- expressionLib:
- |
var calcMem = function calcMem(instanceType) {
var mem_from_instance_type = {
"small" : "64m",
"medium" : "256m",
"large" : "1g",
}
if (!(instanceType in mem_from_instance_type)) {
return "default-mem"
}
return mem_from_instance_type[instanceType];
}
var calcInstanceType = function calcInstanceType(type) {
var instance_type_from_mem = {
"small" : "Standard",
"medium" : "StandardHiCpu",
"large" : "StandardHighIo"
}
if (!(type in instance_type_from_mem)) {
return "Standard"
}
return instance_type_from_mem[type];
}
var calcCoresNumber = function calcCoresNumber(cores) {
var cores_from_type = {
"small" : 2,
"medium" : 3,
"large" : 4
}
if (!(cores in cores_from_type)) {
return 2
}
return cores_from_type[cores];
}
class: InlineJavascriptRequirement
baseCommand:
- echo
inputs:
- label: input size
doc: |
The input size
type:
- "null"
- string
default: 128m
inputBinding:
prefix: --mem
valueFrom: $(calcMem(self))
id: '#input_size'
outputs:
- label: output file
doc: |
The output file
type: File
id: '#output_file'
outputBinding:
glob: stdout.txt
stdout: stdout.txt
successCodes:
- 0
https://schema.org/author:
class: https://schema.org/Person
https://schema.org/name: Sehrish Kanwal
https://schema.org/email: sehrish.kanwal@umccr.org
cwlVersion: v1.1
$schemas:
- https://schema.org/version/latest/schemaorg-current-http.rdf
$namespaces:
s: https://schema.org/
ilmn-tes: http://platform.illumina.com/rdf/ica/
Could this be with your map-resource_requirements_dev-wf
workflow somewhere?
@alexiswl - yes, it's my workflow that I had created for opening upissue with Ilumina. It's on a branch, perhaps I can open up a PR against main to resolve this issue?
No worries, can you link which branch it's on here and I'll close the issue
@alexiswl - I have opened up PR for this tool https://github.com/umccr/cwl-ica/pull/164
Resolved by #164
Incident occurred with https://github.com/umccr/cwl-ica/pull/148 where project.yaml was incomplete with umccrise tool existing in production_workflows but not in project.yaml, this meant
Failed with