add first wdl workflow

[gehadkamel]

Oakvar setup example with wdl

[rkimoakbioinformatics]

email and password should be passed to ov system setup to avoid the program to drop down to an interactive prompt. In the setup yaml file, ov_store_email and ov_store_pw should be defined.

Then, the next question is how to give these two pieces from the cromwell command. Can the wdl file accept command-line arguments for the email and password?

[gehadkamel]

Oakvar run example with wdl.

Regarding email and password, I might dig into cromwell documentation regarding accepting command line inputs.

[rkimoakbioinformatics]

With cromwell, I get the following error:

Process SyncManager-1:
Traceback (most recent call last):
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/managers.py", line 608, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/managers.py", line 154, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/connection.py", line 448, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/connection.py", line 591, in __init__
    self._socket.bind(address)
OSError: AF_UNIX path too long
Traceback (most recent call last):
  File "/home/rick/mambaforge/envs/wdl/bin/ov", line 8, in <module>
    sys.exit(main())
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/__main__.py", line 180, in main
    handle_exception(e)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/__main__.py", line 159, in handle_exception
    raise e
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/__main__.py", line 173, in main
    ret = args.func(args)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/cli/__init__.py", line 5, in change_args_for_cli
    ret = func(args)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/cli/run.py", line 7, in cli_run
    ret = run(args)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/cli/__init__.py", line 32, in run_cli_func
    handle_exception(e)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/__main__.py", line 159, in handle_exception
    raise e
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/cli/__init__.py", line 24, in run_cli_func
    ret = func(args, **kwargs)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/cli/run.py", line 37, in run
    return loop.run_until_complete(module.main())
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/base/runner.py", line 324, in main
    await self.setup_manager()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/site-packages/oakvar/base/runner.py", line 101, in setup_manager
    self.manager.start()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/managers.py", line 583, in start
    self._address = reader.recv()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/home/rick/mambaforge/envs/wdl/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

This error does not happen on your system? In the call-annoation/execution folder, exampleinput.sqlite was produced?

[gehadkamel]

No, such error did not occur to me. Did you run the script in conda environment and with cromwell command?

exampleinput.sqlite was generate in the execution folder. Try running the script in conda environment with cromwell command; As I believe new releases might need other dependencies; However, Conda handles this very well.

[rkimoakbioinformatics]

The issue was https://github.com/broadinstitute/cromwell/issues/3647.

I circumvented the issue on my Ubuntu machine by

task annotation {
    File inputFile
    #Get input file from the generate example task
    #run VC on example 
    command{
        export TMPDIR=/tmp
        ov run ${inputFile} -a clinvar -t vcf -d .
    }
    output{
    File out = stdout()
        File annotation_vcf = "exampleinput.vcf"
        File annotation_sqlite = "exampleinput.sqlite"
    }
}

Since /tmp does not exist on Windows and MacOS I guess, it should be more robust. Suggestions:

• Leave the export command but comment it out.
• Leave a note on the export command for Linux users.
• If you find a way to give a command-line argument for TMPDIR to WDL, the script will be better, but it's optional. The above two will suffice for now.

[gehadkamel]

Okay done; And I will look into command line options as discussed. Thanks a lot.

[gehadkamel]

So I updated the setup workflow with user info input and readme as well; and the optional command is added as well, you might try it on your machine with providing an input json file with the export command and validate it.

[gehadkamel]

so in the new script you find the if(!(modules_dir = String? (Optional)) if the user did not provide any adjustment to such variable in the JSON File .. the yaml_task would not run at all and the workflow would be skipped to oak task, this would be the return message Condition NOT met: '!(modules_dir == "String? (optional)")'. Bypassing conditional section Also the setup file and -f flag are now optional in the ov task command! If the condition is met and the user provided only the path to modules directory as I adjusted the command to be echo "modules_dir: ${m_dir} the yaml task would run and be passed to the ov task and setup file would now be passed.

I ran the code and tested it and it succeeded in both scenarios.