Open mikkonie opened 2 years ago
I've received further reports of this happening a week ago or so. Interestingly, all of these have failed once validation has succeeded and moving the files is about to begin.
Maybe it's possible the iRODS connection is for some reason getting shut down or timing out between these actions within the taskflow? I need to look into this further.
Even if this is a network error, this should return a proper error into the landing zone status. Currently the failure is only reported in timeline. If the landing zone was correctly set to FAILED
, the user could retry later. I will take a look into implementing this once I'm done with all the Taskflow updates.
Furthermore, the user's write access to the landing zone remains disabled. This is not a huge issue, but prevents the user from e.g. adding new files even after the zone status is reset by an admin.
This has been recently happening again, so I updated the original description and will try to look into it for v0.14.2.
I found one possible point where this may happen. If for some reason the flow has not correctly reverted and we hit the end of run_flow()
, the zone status is not updated there while everything else to end executing the flow is done. I added a force updating of the status if not yet set into FAILED
or NOT CREATED
.
I'm still not 100% certain why this could happen, maybe an uncaught iRODS exception in a flow/task which causes us not to revert properly? Or a bug in the "set status to FAILED on revert" task itself?
I'll have to monitor if this change does the trick or if something fancier is still needed.
I'll leave this issue into ongoing
for now and monitor if these problems still occur somewhere.
@ericblanc20 reported a case where a landing zone transfer has failed with the following error mesasge:
Error running async flow: Could not receive server response
UPDATE 2024-02: This has been observed when e.g. the file system on an iRODS resource server is slow/unresponsive/offline. The iRODS connection eventually times out leading into this error. We get a failure in the timeline event, but the zone status remains locked in e.g.
MOVING
. We should be able to update the zone status where we catch the exception for the timeline event.