Delocalization not working when using Structs as outputs

[lmtani]

I could not find on WDL spec something saying that structs could not be used as outputs... so I decide to report here.

This is an example I prepared:

version 1.0

struct Test {
    String name
    File path
}

struct Collection {
    Array[Test] samples
}

task GenerateComplexObject {
    input {
        Int items
    }

    command <<<
        python <<CODE
        import sys
        import os
        import json
        items = []
        for item in range(0, ~{items}):
            name = f"test{item}.txt"
            os.system(f"echo 'some content' >  {name}")
            items.append({"name": f"item-{item}", "path": name})

        with open("results.json", "w") as fh:
            json.dump({'samples': items}, fh)
        CODE
    >>>

    runtime {
        docker: "python:3.8"
        memory: "1 GB"
        cpu: 1
        preemptible: 3
        disks: "local-disk " + 10 + " HDD"
    }

    output {
        Collection results = read_json("results.json")
    }
}

workflow TestStruct {
    input {
        Int items
    }

    call GenerateComplexObject {
        input:
            items=items
    }

    output {
        Collection out = GenerateComplexObject.results
    }
}

When using local backend I have no problem, but when using PAPIv2 (cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory) the files from Test struct (path) do not delocalize.

gsutil ls gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/GenerateComplexObject.log
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/gcs_delocalization.sh
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/gcs_localization.sh
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/gcs_transfer.sh
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/rc
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/results.json
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/script
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/stderr
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/stdout
gs://********/TestaStruct/47bc869c-041f-443d-b0bd-d45a1dd203ff/call-GenerateComplexObject/pipelines-logs/

Is it possible to use structs as intended on the example? I'm using Cromwell 52, do not know if it works in previous versions.

[hkeward]

I am having the same problem.

I think the issue may be here.

This seems to be where we check which outputs are type File or Directory, I think it's perhaps missing File-typed outputs within structs?

[patmagee]

@aednichols this seems like an ugly bug

[aednichols]

@rsasch as bug triager

[rsasch]

I created a bug issue in our Jira issue tracker: https://broadworkbench.atlassian.net/browse/WA-358

[Coppini]

Still experiencing this problem. It seems we cannot use Array[File] inside structs for now.

version development
​
workflow Test {
    input {
        String file_name = "file.txt"
        String file_contents = "teste"
    }
​
    call WriteFile {
        input:
            file_name=file_name,
            file_contents=file_contents
    }
​
    Array[File] array_file = [WriteFile.output_file, WriteFile.output_file]
​
    MultiTypeStruct test_struct = {
        "file_name" : file_name,
        "file" : WriteFile.output_file,
        "array_file" : array_file
    }
​
    output {
        MultiTypeStruct multi_type_struct_test = test_struct
    }
}
​
struct MultiTypeStruct {
    String file_name
    File file
    Array[File] array_file
}
​
task WriteFile {
    input {
        String file_name
        String file_contents
    }
​
    command <<<
        echo -e """~{file_contents}""" > ~{file_name}
    >>>
​
    runtime {
        docker: "gcr.io/google.com/cloudsdktool/cloud-sdk:330.0.0-alpine"
        preemptible: 3
    }
​
    output {
        File output_file = "~{file_name}"
    }
}

You can easily see an error happening when running a simple workflow like this. As long as you have an Array[File] inside a struct, it will keep on failing. In my case, I'm using version development, and the last task on the workflow simply gets stuck with status Running while the workflow itself moves to status Aborting and stays stuck permanently in Aborting (never actually moving its status to Aborted).

Experienced this issue with Cromwell versions 63 and 74, while using GCP lifescience v2 backend.

[yunhailuo]

I don't have access to broad jira so I'm wondering if there is any progress on this bug? @rsasch

We get into the same trouble here. We run our WDL on AWS with batch backend. To share a little more info in addition to what people already see, I saw in the DELOCALIZING OUTPUTS section "reconfigured-script.sh" I noticed it failed to delocalize files in Array[File] in our struct just like what others see. It seems those files are skipped and not "scanned" just like @hkeward pointed out above.