search

radical-cybertools / radical.owms

Tiered Resource OverlaY
Other
0 stars 1 forks source link

File staging does not check whether the file copy is successful #78

Open mturilli opened 10 years ago

mturilli commented 10 years ago

I had file staging failing in both directions due to quota exceeded and disk full. In both cases, file staging did not fail. Only once, troy froze because the target disk was full. Other times, it just went on like the transfer was successful.

andre-merzky commented 10 years ago

This is a known saga-python level problem: sftp is not returning error codes, and thus we have to parse the sftp output for error messages. Alas, sftp is chatty, and output differs from version to version, and from error to error -- so you probably stumbled over a combination we do not yet catch.

Would you be able to reproduce the error with SAGA_VERBOSE=DEBUG, and look for the respective sftp error output? That would help me to fix that. Alternatively, on what combination of machines did that happen (source/target)? I can try to fill my quota on the target ;)

Thanks!

mturilli commented 10 years ago

Hi Andre,

Thank you for the details. File staging was done between ip-10-195-187-47 and stampede. To replicate the event I will need to fill up root at ip-10-195-187-47. Let me finish with this round of experiments and then I will work on reproducing the issue with the saga logs cranked up.

Meanwhile, why not using paramiko or twisted to do the file transfer without having to grab the output of sftp?

Best!