search

irods / irods_demo

BSD 3-Clause "New" or "Revised" License
6 stars 14 forks source link

`irods-client-s3-api` service fails to start server #85

Open alanking opened 1 week ago

alanking commented 1 week ago

Using tip of main, I get the following output:

$ docker compose up irods-client-s3-api
[+] Running 5/4
 ✔ Network irods-demo_default                     Created    0.1s
 ✔ Container irods-demo-minio-1                   Created    0.1s 
 ✔ Container irods-demo-irods-catalog-1           Created    0.1s 
 ✔ Container irods-demo-irods-catalog-provider-1  Created    0.1s 
 ✔ Container irods-demo-irods-client-s3-api-1     Created    0.0s 
Attaching to irods-demo-irods-client-s3-api-1
irods-demo-irods-client-s3-api-1  | [2024-11-12 20:02:37.213] [P:1] [info] [T:1] Initializing server.
irods-demo-irods-client-s3-api-1  |  ERROR: [-] /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  |  ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  | Error: USER_SOCK_CONNECT_ERR: Connect error
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1 exited with code 0

I tried updating to 0.3.0...

diff --git a/docker-compose.yml b/docker-compose.yml
index 133cf2c..e96fb45 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -52,7 +52,7 @@ services:
                 condition: service_healthy

     irods-client-s3-api:
-        image: irods/irods_s3_api:0.2.0
+        image: irods/irods_s3_api:0.3.0
         volumes:
             - ./irods_client_s3_api/config.json:/config.json:ro
         ports:

...and got a bit more:

irods-demo-irods-client-s3-api-1  | [2024-11-12 20:07:11.511] [P:1] [info] [T:1] Initializing server.
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: [-]   /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: [-]   /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: [-]   /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: [-]   /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: [-]   /irods_source/lib/core/src/sockComm.cpp:863:int connectToRhost(rcComm_t *, int, int) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |     [-]     /irods_source/lib/core/src/sockComm.cpp:611:irods::error readVersion(irods::network_object_ptr, version_t **) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |             [-]     /irods_source/lib/core/src/sockComm.cpp:229:irods::error readMsgHeader(irods::network_object_ptr, msgHeader_t *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [failed to call 'read header']
irods-demo-irods-client-s3-api-1  |                     [-]     /irods_source/plugins/network/src/tcp.cpp:186:irods::error tcp_read_msg_header(irods::plugin_context &, void *, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  |                             [-]     /irods_source/plugins/network/src/tcp.cpp:67:irods::error tcp_socket_read(int, void *, int, int &, struct timeval *) :  status [SYS_SOCK_READ_ERR]  errno [Connection reset by peer] -- message [error reading from socket after [0] bytes read]
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  |  ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | remote addresses: 192.168.64.4 ERROR: _rcConnect: connectToRhost error, server on irods-catalog-provider:1247 is probably down status = -116104 SYS_SOCK_READ_ERR, Connection reset by peer
irods-demo-irods-client-s3-api-1  | Error: USER_SOCK_CONNECT_ERR: Connect error
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1  | 
irods-demo-irods-client-s3-api-1 exited with code 0
alanking commented 1 week ago

I tried restarting the service and saw the server come up and run normally. It's only the initial attempt that fails, it seems.

trel commented 1 week ago

and noting here, intiial and restarted both connect fine for me. so we're not yet sure how to trap this or what it means.

trel commented 1 week ago

works for me:

$ docker compose up irods-client-s3-api
[+] Running 4/4
 ✔ Container irods-demo-minio-1                   Created                                                                                                        0.1s 
 ✔ Container irods-demo-irods-catalog-1           Created                                                                                                        0.1s 
 ✔ Container irods-demo-irods-catalog-provider-1  Created                                                                                                        0.0s 
 ✔ Container irods-demo-irods-client-s3-api-1     Created                                                                                                        0.0s 
Attaching to irods-client-s3-api-1
irods-client-s3-api-1  | [2024-11-13 00:59:41.899] [P:1] [info] [T:1] Initializing server.
irods-client-s3-api-1  | [2024-11-13 00:59:42.192] [P:1] [info] [T:1] Server is ready.
alanking commented 1 week ago

Another update... I don't even have to restart the service myself. The restart directive in the Compose file causes the service to come up successfully all by itself. I only know this because I run this to follow the output after docker compose up returns my prompt:

docker compose logs -f irods-client-s3-api

So... in the end, the service is running, but it just fails the first time for some reason.

trel commented 1 week ago

nothing more than ... because the irods provider wasn't yet ready?

trel commented 1 week ago

the initial connection is required to standup the s3 api... but then.. i believe it will gracefully reconnect in case the provider 'goes away' and comes back...