dib-lab / elvers

(formerly eelpond) an automated RNA-Seq workflow system
https://dib-lab.github.io/elvers/
Other
28 stars 3 forks source link

Further improvements to DAG handling #73

Closed charlesreid1 closed 5 years ago

charlesreid1 commented 5 years ago

Pull request to expand #69 to handle three directed acyclic graph flags:

These flags all turn on the --dry-run flag.

To accomplish the second two flags, I implemented a context manager that will capture stdout when the user is creating the directed acyclic graph, and either prints that captured output to the screen (if the user passes the --dag flag, keeping that flag working as @bluegenes implemented it in #69), or prints the captured output to a file (if the user passes the --dagfile flag), or uses dot to pass the captured output to dot and convert it into a png file (if the user passes the --dagpng flag).

charlesreid1 commented 5 years ago

Example commands:

./run_eelpond examples/nema.yaml default --dag > dag.dot
./run_eelpond examples/nema.yaml default ---dagfile dag.dot
./run_eelpond examples/nema.yaml default --dagpng dag.png
charlesreid1 commented 5 years ago

TODO: add tests

charlesreid1 commented 5 years ago

Oh, and hey, look at that - a blog post from May 2014 on context managers, which we used in this pull request, from @camillescott - http://www.camillescott.org/2014/05/05/context-management/

charlesreid1 commented 5 years ago

Good idea! I went ahead and made those changes.

charlesreid1 commented 5 years ago

Before I implement a test for this, I just wanted to make some notes here about the eelpond testing approach.

I just noticed the tests you're running are in the .travis.yml file but not in a script or a test directory. Thinking about this CaptureStdout object, it would actually allow you to do something really cool - you can wrap the tests you are currently running from the shell (via .travis.yml) in a python script (a unit test) that could be run with pytest. In other words, if you want to run a shell command like ./run_eelpond arg1 arg2 you could write a unit test that calls that command from Python using subprocess.call(['./run_eelpond','arg1','arg2']).

If you then use the context manager approach to capture stdout from that subprocess call that would allow capturing the output from the Snakemake test (edit: on second thought you may be able to do this with subprocess's PIPE and not need to bother with the context manager) - and check it for specific text, rather than simply checking that it doesn't crash. (This is useful if you have a workflow that may go several different routes, and you want to not just verify the workflow is running but make sure it is following the right logical path).

I'll put together a unit test for the new --dag flags shortly - everything mentioned above will go in a different pull request - but just wanted to to jot down these thoughts somewhere.