search

irods / python-irodsclient

A Python API for iRODS
Other
63 stars 73 forks source link

Have PRC keep the connection open during checksum calculation #564

Closed luijs closed 2 months ago

luijs commented 3 months ago

During transfer of large files(100's of GB's) with checksumming enabled, the PRC can break the connection when the checksum is being calculated. Is there a way for the PRC to see this and make sure the connection stays open?

See https://groups.google.com/g/irod-chat/c/k7Y7UXdXJ5Y for detailed discussion

Regards,

Joris

edit: To be a bit more complete, seen with: irods 4.3.1 Ubuntu 20.04 PRC 2.0.1

mstfdkmn commented 3 months ago

To contribute: we had the same or similar issue reported at and I supposed this was fixed with this after https://github.com/irods/python-irodsclient/issues/377#issuecomment-1487986787.

korydraughn commented 3 months ago

@luijs Is this the same issue we discussed / worked-out during the UGM troubleshooting session (i.e. it was fixed by upgrading)?

korydraughn commented 3 months ago

I just read the google chat discussion. Seems the upgrade just revealed more information about issue.

d-w-moore commented 2 months ago

@luijs - I now have verified from experimentation that a patch from #570 (a fix for #569), along with an assignment to the connection timeout : 

s.connection_timeout = None

should do the trick of preventing the NetworkException during large (eg >=100G) puts with checksum, as also with just calling session.data_objects.chksum( ...)on those large objects with the verifyChecksum option enabled.
So, I'm pretty much ready to close this issue and consider it solved as soon as #570 is merged.

With that knowledge, please let us know if you have further input on this one.

alanking commented 2 months ago

Resolved by changes in https://github.com/irods/python-irodsclient/pull/570.