snowflakedb / snowflake-ingest-java

Java SDK for the Snowflake Ingest Service -
http://www.snowflake.net
Apache License 2.0
64 stars 51 forks source link

Infinite retries if no internet connection #762

Open berl-a opened 1 month ago

berl-a commented 1 month ago

When there's no connection, Snowflake library goes into infinite loop, trying to reconnect to the database forever. Part of logs:

15:21:26.024 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: Connection reset 15:21:26.025 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:27.027 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: No such host is known (xxx.us-east-1.snowflakecomputing.com) 15:21:27.027 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:29.029 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:29.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:33.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:33.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:37.032 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:37.032 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:41.034 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: No such host is known (When there's no connection, Snowflake library goes into infinite loop, trying to reconnect to the database forever. Logs: 15:21:26.024 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: Connection reset 15:21:26.025 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:27.027 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: No such host is known (xxx.us-east-1.snowflakecomputing.com) 15:21:27.027 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:29.029 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:29.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:33.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:33.030 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443 15:21:37.032 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - I/O exception (java.net.UnknownHostException) caught when processing request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443: xxx.us-east-1.snowflakecomputing.com 15:21:37.032 [] [] [pool-12-thread-1] INFO n.s.c.j.i.a.h.i.execchain.RetryExec - Retrying request to {s}->https://xxx.us-east-1.snowflakecomputing.com:443

berl-a commented 1 month ago

Libraries used: Gradle: net.snowflake:snowflake-jdbc:3.13.30 Gradle: net.snowflake:snowflake-ingest-sdk:2.0.3

I'd expect the system to retry only as many times as it says in the maxRetryCount, i.e. 25