wtsi-npg / baton

iRODS client programs and API
http://wtsi-npg.github.io/baton
GNU General Public License v2.0
19 stars 19 forks source link

Improve iRODS connection timeout/refresh #262

Closed kjsanger closed 2 years ago

kjsanger commented 2 years ago

Improve connection refresh for the case when the client is running, but blocked on STDIN, waiting for the next JSON document. Previously, any open iRODS connection would remain so until the next document could be read. By adding a separate connection timeout management thread, the connection can be closed promptly, even when blocked on reading. A mutex protects the connection from being accessed by the main and timeout management threads simultaneously.

kjsanger commented 2 years ago

I think this is all fixed now. Testing a slow feed of inputs with a short timeout:

cat <(for i in {1..50} ; do sleep 2 && cat test.json ; done) |\
   ./src/baton-list --connect-time 10 --verbose
2022-08-01T12:16:30Z NOTICE Opening a new iRODS connection
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
2022-08-01T12:16:38Z NOTICE Closed the iRODS connection after a timeout of 10 seconds
2022-08-01T12:16:40Z NOTICE Opening a new iRODS connection
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
{"collection": "/testZone/home/irods", "data_object": "10G_zero"}
2022-08-01T12:16:48Z NOTICE Closed the iRODS connection after a timeout of 10 seconds
2022-08-01T12:16:50Z NOTICE Opening a new iRODS connection

etc.
kjsanger commented 2 years ago

That's great, thanks. I had a crack at it with valgrind too and wasn't able to find problems other than those you mentioned.