coam-on-stampede file staging error

[andre-merzky]

The relevant portion of the log reads:

2015:03:25 10:34:09 radical.pilot.MainProcess: [DEBUG   ] write: [   20] [  153] (mput  "/home/merzky/coam-on-stampede/penta.crd" "/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"\n)
2015:03:25 10:34:09 radical.pilot.MainProcess: [DEBUG   ] read : [   20] [  156] (mput  "/home/merzky/coam-on-stampede/penta.crd" "/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"\n)
2015:03:25 10:34:09 radical.pilot.MainProcess: [DEBUG   ] read : [   20] [  157] (Uploading /home/merzky/coam-on-stampede/penta.crd to /work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd\n)
2015:03:25 10:34:10 radical.pilot.MainProcess: [DEBUG   ] read : [   20] [  152] (remote open("/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"): No such file or directory\nsftp> )
2015:03:25 10:34:10 radical.pilot.MainProcess: [ERROR   ] DoesNotExist: file copy failed: mput  "/home/merzky/coam-on-stampede/penta.crd" "/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"
Uploading /home/merzky/coam-on-stampede/penta.crd to /work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd
remote open("/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"): No such file or directory
sftp> 
An error occurred: file copy failed: mput  "/home/merzky/coam-on-stampede/penta.crd" "/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"
Uploading /home/merzky/coam-on-stampede/penta.crd to /work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd
remote open("/work/01083/tg803521/radical.pilot.sandbox/pilot-551280f374df923b2818963b/staging_area/iter0/penta.crd"): No such file or directory
sftp>  (/home/merzky/ExTASY-tools/local/lib/python2.7/site-packages/saga/utils/pty_shell.py +1010 (run_copy_to)  :  raise se.DoesNotExist._log (info['logger'], "file copy failed: %s" % str(out)))
Closing session, exiting now ...

Note that penta.crd has already been staged into [pilot_sandbox]/staging_area/penta.crd at this point -- I am not sure why another attempt at staging occurs? Also, w/o looking at the code, could it be that the CREATE_PARENTS flag on the RP staging directive is missing (if that directive is used, dunno)?

[vivek-bala]

Yes, this transfer is done using directives : https://github.com/radical-cybertools/ExTASY/blob/master/src/radical/ensemblemd/extasy/bin/Preprocessor/Amber/preprocessor.py#L40-L44.

Isn't the CREATE_PARENTS flag the default behaviour ?

[ashkurti]

Just noticed that this is a similar error as I noticed today and explained in Issue #150

[vivek-bala]

I think this should be fixed in devel, could you try again please?

[ashkurti]

Thanks for this but the same errors persisted for the CoCo / amber workflow on ARCHER even after reinstalling a fresh environment using the devel branch for the ExTASY installation.

[ExTASY-tools] ardita@poirot 248% python -c 'import radical.ensemblemd.extasy as extasy; print extasy.version'
0.1.3.1-beta-11-gd083439

The radical folder in stampede, public now, can be accessed at /work/02998/ardi/radical.pilot.sandbox/pilot-55144fdcf8cdba2501d24e69

The extasy log can be found at: https://gist.github.com/ashkurti/b4518eabd1a034788baa

[vivek-bala]

I was able to reproduce this error. I get this error when I install RP from pip. But when I install RP from the master branch, it works fine and the parent folder(s) are created. Could you confirm as to which Radical Pilot you used ?

[ashkurti]

I admit I used the pip install before, but I have tried this morning (installing radical pilot from the master branch of its repository) at least twice and I still do get problems. I believe it is a different error now (See https://gist.github.com/ashkurti/f874e3c1b175203eec80#file-gistfile1-txt-L2189).

The log can be found at https://gist.github.com/ashkurti/f874e3c1b175203eec80 whereas the radical.pilot folder can be publicly accessed in stampede at /work/02998/ardi/radical.pilot.sandbox/pilot-55153a9cf8cdba61f60b09ce

[andre-merzky]

Master indeed contains a couple of commits which have been done as small bug fixes. https://github.com/radical-cybertools/radical.pilot/commit/2ae27f8e4cff2759ca84db4ed447f529c1b2c615 is a commit which is in master, but not the released version, and looks like a good explanation for the problem. I would not mind doing a quick release of RP from master at that stage, if the EnMD folx would like to have that in pypi.

Scroll down on https://github.com/radical-cybertools/radical.pilot/compare/v0.23...master to see a full diff of master and last release.

[oleweidner]

Hi all, do we have a solution for this ticket?

For the sake of beta-testing - would it be feasible to release a new version of RP from master?

[andre-merzky]

Yes, this would be the proposal. If you'd agree, I can do this today. Please give master a quick try (you need to do your testing anyway ;)? See the diff above, and check if that looks ok to you. If all is fine, I can push that to pypi today.

[andre-merzky]

Ole, Vivek -- any feedback of we can/should push RP master to pypi? I rather not do that before you explicitly ok it... thanks!

[oleweidner]

I'm testing with the RP 0.24 release right now -- so far so good ;-)

[vivek-bala]

This ticket was handled via email communications so I guess this ticket can be closed.