search

EliLillyCo / pytest-wdl

WDL plugin for pytest
https://pytest-wdl.readthedocs.io
Apache License 2.0
48 stars 8 forks source link

Directories as inputs for tasks #135

Closed ghost closed 4 years ago

ghost commented 4 years ago

Hey,

I started using pytest-wdl as my workflows get to long to run them everytime I make changes just to test that everything is accurate. I appreciate it very much. However, I do not get it to run with a directory as input. For one task I cannot specify the files specifically that I receive as an output and need to use the whole directory as an input for the next task. The problem is, it fails with:

failed :: error: "InputError", message: "inpnot found: /private/var/folders/hx/bv0hfckd64x0wd90_mg7yngm0000gn/T/tmp96wbc67o/test"

The directory test is located under data with the other inputs and expected outputs. Is there any way to make it work?

Just to give you some background information. If I run bcl2fastq and interop tools, it expects a directory as input. I cannot specify individual files. These tools are rather common in NGS analyses and cromwell can handle directories quite well. It would be nice to test these tasks as well.

ghost commented 4 years ago

I found the problem. The function add_files in runtime/task.py has an explicit check for files. It fails for directories. Would it be possible to allow directories as input?

for host_file in host_files:
  if host_file not in self.input_file_map:
    if not os.path.isfile(host_file):
      raise Error.InputError("input file not found: " + host_file)
        host_files_by_dir.setdefault(os.path.dirname(host_file), set()).add(host_file)
jdidion commented 4 years ago

Hi there - it the code you're looking at there is in miniwdl, which is a separate library. You can submit your feature request there. But just be aware that as of WDL 1.0, directory inputs are not supported. They will be supported in 2.0 with the introduction of the Directory type, at which point it should be supported by all the various workflow engines.