search

radical-cybertools / ExTASY

MDEnsemble
Other
1 stars 1 forks source link

COCO/Amber CU failed due to I/O error during output staging #156

Closed ibethune closed 9 years ago

ibethune commented 9 years ago

I have seen several errors of this sort this morning, where a CU fails due to an error during the output staging step. The log contains a message like:

2015:03:30 12:02:25 radical.pilot.MainProcess: [DEBUG ] flush: [ 14] [ ](flush pty read cache) 2015:03:30 12:02:25 radical.pilot.MainProcess: [DEBUG ] flush: [ 18] [ ](flush pty read cache) 2015:03:30 12:02:25 radical.pilot.MainProcess: [DEBUG ] write: [ 18] [ 119](ls /work/e290/e290/e290ib/radical.pilot.sandbox/pilot-55192c88d7bf75a56b1d81fa/unit-55192d32d7bf75a56b1d8206/md0.ncdfn) 2015:03:30 12:02:25 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 157](ls /work/e290/e290/e290ib/radical.pilot.sandbox/pilot-55192c88d7bf75a56b1 ^Hd81fa/unit-55192d32d7bf75a56b1d8206/md0.ncdfnCouldn't send packet: Broken pipen) 2015:03:30 12:02:25 radical.pilot.MainProcess: [ERROR ] Output transfer failed: read from process failed '[Errno 5] Input/output error' : (B/s --:-- ETA/home/h012/ibethune/testcoco11/penta.crd 100% 2165 2.1KB/s 00:00 sftp> ls /work/e290/e290/e290ib/radical.pilot.sandbox/pilot-55192c88d7bf75a56b1 ^Hd81fa/unit-55192d32d7bf75a56b1d8206/md0.ncdf Couldn't send packet: Broken pipe

Full set of logs, and the pilot directory are in /work/e290/e290/shared/iain/io-error/ on ARCHER

vivek-bala commented 9 years ago

Might be the same as one of the old tickets for archer. I believe the fix was to set $HOME/saga.cfg as

[saga.utils.pty]
ssh_share_mode = no
ibethune commented 9 years ago

Should that go into $HOME on the box where 'extasy' is run, or on the execution machine?

vivek-bala commented 9 years ago

This would be on the local machine (laptop/vm), where extasy is run.

ibethune commented 9 years ago

Thanks, but no, that doesn't solve the problem in this case.

andre-merzky commented 9 years ago

Iain -- please note that should be $HOME/.saga.cfg -- ie. a dot-file...

ibethune commented 9 years ago

Ah OK, I will try again with that!

oleweidner commented 9 years ago

Is this documented somewhere in the ExTASY docs and if so, does the documentation show the "dot"-file correctly?

ibethune commented 9 years ago

Closing, fixes by the .saga.cfg file (or env var)