Skeleton/Synapse integration

[mturilli]

• Fork Skeleton repository
• Implement 'Synapse' connector based on the shell connector.
• Implement the translation of the Skeleton parameter into synapse parameters within the Synapse code base.
• Create a pull request for Skeleton so to make the support of Synapse 'official' (only if no/low overhead)

[mturilli]

Ming and I can work around #3 but this is a road block to start the experiments. If this is proving > 6 hours-long implementation, would just integrating Synapse with aimes.emgr be a more rapid implementation?

[andre-merzky]

Hey Matteo, Ming,

I don't believe its a 6-hour coding thing. The problem is, as usual, to allocate time in the first place... :/ But thanks for the heads-up wrt. priorities! Let me attack this tonight, and sync tomorrow after the RP call to see what the status is? By then I should either have a fair estimate on the effort, and ideally some initial results. If not we should reshuffle....

[andre-merzky]

Quick update: the synapse side work is done: synapse can now be busy for a certain amount of time, and can read/write from/to specific files (instead of the default location).

What remains is to move that to a skeleton task.c replacement. A stub for that exists, it needs filling out with code, where that code can be mostly hijacked from radical-synapse-sample.

so I think I can finish this off later today, but likely only after the RP call (have to break for now).

[andre-merzky]

Ok, the stuff is now implemented in aimes.skeleton, in the feature/synapse_tasks branch. You will need the feature/named_storage branch from radical.synapse, and devel from radical.utils, and then should see something like shown in the commit message here:

https://github.com/applicationskeleton/Skeleton/commit/38c3aac8b707007dd7945f21693c28b3cc0d0900

So you should be able to replace task with task.py in the slkeleton workloads.

Could you give it a try? Some cleanup is needed -- specifically wrt. installation and skeleton script generation...

[mturilli]

Great, thank you Andre! Ming: care to give it a try? Happy to help/contribute if needed.

[mingtaiha]

Hey, I was able to change the CU description from task to task.py. On XSEDE machines, I find that I need to install Synapse. How long would 'pip install --upgrade radical.synapse' take.

Also, when I run on OSG, I get the error that the executable task.py is not found. I added the proper folder to $PATH on the login node and gave it the right permissions. Is the PATH variable different for different nodes on OSG?

[andre-merzky]

Hey Ming,

on the XSEDE machines, we found that manual synapse install is the most practical approach. You would need to do something like this on all target machines (which is boring, but oterwise painless I hope):

ssh stampede
virtualenv $HOME/ve.synapse
. $HOME/ve.synapse/ve/bin/activate
pip install radical.synapse
pip install aimes.skeleton

and then in the CU descrption, add . $HOME/ve.synapse/bin/activate as pre_exec command. That should make synapse and skeleton's task.py available.

One caveat: the pip commands will pull the releases from pypi, which does not yet contain the synapse/skeleton integration. Once you have done some preliminary testing with the branches I listed, I'm happy to quickly push out a radical.synapse release, and will talk to Dan to do something similar for aimes.skeleton.

[mturilli]

Thanks Andre. What do we do with OSG? Still working on that one?

[andre-merzky]

Yes, still working on the deployment there...

[mingtaiha]

I was able to run it on my local machine. As discussed, I ran ./bin/aimes-skeleton-generate -m shell -c -o test.sk -i src/aimes/skeleton/sample-input/sample.input and was was able to emulate it correctly. I also used bag.input as input, but I get io read failed: File exists

[andre-merzky]

At what point exactly did you get the error? Thanks!

[mingtaiha]

I get this error after for each line of test.sk generated using bag.input as input for task.py

[andre-merzky]

Got it. I pushed a fix to synapse which makes it more resilient against partial reads. that is what we saw here: the file was not large enough to fill the buffer on the last step.

[mingtaiha]

I just gave it a run with the fix and it seems to work fine now. I will be waiting for the new release!

[andre-merzky]

great! I am going to close this ticket then, if you don't mind. Please open new tickets for synapse or here, as needed.