search

snowflakedb / snowflake-ingest-java

Java SDK for the Snowflake Ingest Service -
http://www.snowflake.net
Apache License 2.0
65 stars 51 forks source link

Refresh stage metadata automatically in the background to avoid exceptions encountered during file upload in uploadWithoutConnection #753

Open ademfreshbooks opened 3 months ago

ademfreshbooks commented 3 months ago

Snowflake Ingest SDK: 2.1.0

For long running clients in streaming mode, It seems ~ every hour, session expires for StreamingIngestStage (used by FlushService), that raises:

Exception encountered during file upload in uploadWithoutConnection

Then StreamingIngestStage retries request again, by default 5 times.

Usually this does the job, however for app observability sake, this is not perfect as this creates false positives.

Also, there does not appear to be any configuration options available to avoid this behaviour, or options to override some of the things.

Can you please try to add auto refresh for stage metadata to avoid this situation?

sfc-gh-lsembera commented 2 months ago

Thanks for the suggestion, @ademfreshbooks. We do attempt to auto-refresh stage metadata when uploads start failing, but we can improve the situation by not logging the initial exception when metadata was immediately successfully refreshed.

ademfreshbooks commented 2 months ago

Thanks @sfc-gh-lsembera

Upon deeper inspection, it seems the error is being logged in net.snowflake.client.jdbc.SnowflakeFileTransferAgent here before it is being thrown to upper layer.

So it seems Snowflake Ingest SDK is already doing what you say, ie not logging first exception as error as shown here.

What would be your recommendation how to address this behaviour?

It would be ideal to have flag to disable this particular log call...

sfc-gh-lsembera commented 2 months ago

Yeah, the SDK doesn't log the first upload error, but the library we are using to upload data does. Thanks for reporting the issue, we will post updates here once the issue gets prioritized.