chanzuckerberg / miniwdl

Workflow Description Language developer tools & local runner
MIT License
174 stars 54 forks source link

Standard library function `as_map` outputs EvalError if File type is encountered #700

Open stxue1 opened 3 months ago

stxue1 commented 3 months ago

The function as_map is unable to handle cases where one of its nested types is File.

version 1.1

workflow test_as_map {
  input {
    Array[Pair[String, File]] test_arr = [("a.bam", "a.bai")]
  }

  output {
    Map[String, File] o = as_map(test_arr)
  }
}
2024-07-02 10:56:52.474 wdl.w:test_as_map workflow start :: name: "test_as_map", source: "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_as_map.wdl", line: 3, column: 1, dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_105652_test_as_map"
2024-07-02 10:56:52.476 wdl.w:test_as_map miniwdl :: version: "v1.12.0", uname: "Linux pop-os 6.9.3-76060903-generic #202405300957~1718348209~22.04~7817b67 SMP PREEMPT_DYNAMIC Mon J x86_64"
2024-07-02 10:56:52.492 wdl.w:test_as_map workflow test_as_map (/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_as_map.wdl Ln 3 Col 1) failed :: dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_105652_test_as_map", error: "EvalError", node: "output-o", pos: {"source": "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_as_map.wdl", "line": 9, "column": 27}
2024-07-02 10:56:52.493 wdl.w:test_as_map aborting workflow
2024-07-02 10:56:52.493 miniwdl-run  :: error: "EvalError", node: "output-o", pos: {"source": "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_as_map.wdl", "line": 9, "column": 27}, dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_105652_test_as_map"

Switching the File type to String, the workflow is able to run:

version 1.1

workflow test_as_map {
  input {
    Array[Pair[String, String]] test_arr = [("a.bam", "a.bai")]
  }

  output {
    Map[String, String] o = as_map(test_arr)
  }
}
{
  "dir": "/home/heaucques/Documents/wdl-conformance-tests/20240702_105449_test_as_map",
  "outputs": {
    "test_as_map.o": {
      "a.bam": "a.bai"
    }
  }
}

It seems like whenever as_map sees a File, an EvalError is outputted.

There is also a unit test in the WDL spec that depends on this behavior, and this is preventing miniwdl from passing it.

stxue1 commented 3 months ago

The same issue appears with the collect_by_keys function. Running the given unit test in the WDL spec:

2024-07-02 12:48:16.161 wdl.w:test_collect_by_key workflow start :: name: "test_collect_by_key", source: "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_collect_by_key.wdl", line: 3, column: 1, dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_124816_test_collect_by_key"
2024-07-02 12:48:16.163 wdl.w:test_collect_by_key miniwdl :: version: "v1.12.0", uname: "Linux pop-os 6.9.3-76060903-generic #202405300957~1718348209~22.04~7817b67 SMP PREEMPT_DYNAMIC Mon J x86_64"
2024-07-02 12:48:16.186 wdl.w:test_collect_by_key workflow test_collect_by_key (/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_collect_by_key.wdl Ln 3 Col 1) failed :: dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_124816_test_collect_by_key", error: "EvalError", node: "output-is_true2", pos: {"source": "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_collect_by_key.wdl", "line": 20, "column": 24}
2024-07-02 12:48:16.186 wdl.w:test_collect_by_key aborting workflow
2024-07-02 12:48:16.186 miniwdl-run  :: error: "EvalError", node: "output-is_true2", pos: {"source": "/home/heaucques/Documents/wdl-conformance-tests/unit_tests/test_collect_by_key.wdl", "line": 20, "column": 24}, dir: "/home/heaucques/Documents/wdl-conformance-tests/20240702_124816_test_collect_by_key"

MiniWDL crashes when calling collect_by_key on a WDL type with nested files.