fhcrc / nestly

Make and use nested directory trees corresponding to combinatorial choices of parameters
http://nestly.readthedocs.org
MIT License
21 stars 2 forks source link

Possible to allow nests to diverge at a given level? #22

Open cerebis opened 9 years ago

cerebis commented 9 years ago

I've been using Nestly and Scons to implement a simulation tool which performs sweeps. I've reached a point where I wish to test different algorithms, each of which have their own distinct parameter set. I was thinking I could do this easily in Nestly but it seems it would be paddling upstream. Perhaps I am missing something?

Say at level N in the tree, you have a bifurcation to A and B, and that under this point, A and B will have a different topology. Is that possible to define paths within the Nest without introducing a lot of "if then" to check the state of the branch from inside my target functions?

metasoarous commented 9 years ago

Afraid not. One thing that might prove helpful though is that you can create multiple nest wrappers around an scons environment. So you could actually created separate nest structures within each of A B that way. But it's some extra indirection an mind bending, so you'll have to decide whether it's worth the while or not.

cerebis commented 9 years ago

Thanks for the response. I had thought of doing exactly that (separate nests) but I think I will stop short of trying to integrate them all. I'll simply chop off these divergent branches and begin them anew independently.

metasoarous commented 9 years ago

Yeah, the slightly complicated thing with separate nests is that you have to do a little bit of extra work to get out any file targets you need from the nested nests (what is this, Inception?)

metasoarous commented 9 years ago

Oy... just realized there is an easier way to do this.

Instead of creating a logical nest with a separate branch for each of the things that are going to have completely different structure, you can sort of simulate one by creating a bunch of separate nests, and then pop back out of them. I think this is a lot cleaner, since it keeps things at the "top level" syntactically, and still let's you aggregate and what not:

# possibly some previous nesting, setup, etc
...

# First create an aggregator so we can pull stuff out 
n.add_aggregate('stuff_I_care_about', dict)

# The first of our irreconcilable branches
n.add('branch1', ('branch1',))

@n.add_target()
def some_target(outdir, c):
    target = env.Command(<code-furiously>, ...)
    c['stuff_I_care_about']['branch1'] = target
    return target

# More stuff in branch 1
...

# Pop out of branch 1 once done
n.pop('branch1')

# Now create our branch two, add aggregate objects there, etc
n.add('branch2', ('branch2',))

@n.add_target()
def some_other_target(outdir, c):
    <more-furious-coding>

...

n.pop('branch2')

# For possibly many branches
...

# Then do something with our aggregate targets
@w.add_target()
def something_with_aggregate(outdir, c):
    return env.Command('target', c['stuff_I_care_about'].values(), 'some sh call')

Definitely cleaner than having to actually create new nest objects. When I suggested that I was thinking about a situation where there was some other constraint for which I had to create new nests within existing nests, but that doesn't apply here.

Regarding whether or not this is "paddling upstream", it really depends on the situation. I think if you get "enough" usage from any upstream up downstream nesting, then it's fine. But if the only calls to add and pop are of the form above, then you probably would be better off just using vanilla SCons. The only caveat I'll add to that opinion is that I have frequently started projects for which I though nestly would be overkill, and then ended up rewriting with nestly as things developed. You know your work better than anyone though.