search

Factual / drake

Data workflow tool, like a "Make for data"
Other
1.48k stars 110 forks source link

Add directive to support running included scripts in their own working dir #184

Open chen-factual opened 9 years ago

chen-factual commented 9 years ago

For example, if I have a project I've divided into chunks:

overall.d
sub-workflows/
  foo/
    foo.d
    foo.sh
  bar/
    bar.d
    bar.sh

And the top level overall.d %includes foo.d and bar.d. Then inside foo.d, to run foo.sh in a step, I must say sub-workflows/foo/foo.sh. In other words, it's impossible to implement the submodule foo as its own modular piece; it must know about the top level project and that it lives in the subdirectory sub-workflows/foo.

In this case, it's not so bad, because at least they're all in the same project. Let's look at the multiple project case. If I had a proj_a.d in project A try to include a proj_b.d in project B, proj_b.d cannot reference any scripts in project B unless it has knowledge of it's relative path to project A, since proj_a.d'd directory is the working directory.

The implication here is that Drake workflows can never be built to be truly modular pieces that can be composed together by other Drake workflows.

I propose a new % directive, similar to %include, except included workflows maintain their working directories. I've discussed with @amalloy and convinced him of the need for this, and he is tentatively not against it, provided it is possible to implement given JVM constraints. Brief research shows the JNR POSIX project might get us what we need. There is a call to libc's chdir that looks like it might actually change the working directory. It looks like it's at least worth a try.

artstwolf commented 9 years ago

This would be helpful for my projects as well.