search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.32k stars 4.16k forks source link

destination-snowflake: Thread cleanup issue #27741

Closed sivankumar86 closed 1 year ago

sivankumar86 commented 1 year ago

Connector Name

destination-snowflake

Connector Version

1.0.5

What step the error happened?

During the sync

Revelant information

Airbyte version : 0.50.4

Relevant log output

destination > WARN i.a.i.b.IntegrationRunner(watchForOrphanThreads):278 The main thread is exiting while children non-daemon threads from a connector are still active.
Ideally, this situation should not happen...
Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
The main thread is: main (RUNNABLE)
 Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
        at io.airbyte.integrations.base.IntegrationRunner.dumpThread(IntegrationRunner.java:309)
        at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:282)
        at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:155)
        at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
        at io.airbyte.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:87)
        at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationRunner.main(SnowflakeDestinationRunner.java:18)
Log4j2Appender says: WARN i.a.i.b.IntegrationRunner(watchForOrphanThreads):290 Active non-daemon thread: pool-6-thread-2 (RUNNABLE)
 Thread stacktrace: java.base@17.0.4.1/sun.nio.ch.Net.poll(Native Method)
        at java.base@17.0.4.1/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:181)
        at java.base@17.0.4.1/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:285)
        at java.base@17.0.4.1/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
        at java.base@17.0.4.1/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
        at java.base@17.0.4.1/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
        at java.base@17.0.4.1/java.net.Socket$SocketInputStream.read(Socket.java:966)
        at java.base@17.0.4.1/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
        at java.base@17.0.4.1/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
        at java.base@17.0.4.1/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
        at java.base@17.0.4.1/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1460)
        at java.base@17.0.4.1/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1064)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at app//net.snowflake.client.jdbc.internal.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at app//net.snowflake.client.jdbc.internal.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at app//net.snowflake.client.jdbc.RestRequest.execute(RestRequest.java:175)
        at app//net.snowflake.client.core.HttpUtil.executeRequestInternal(HttpUtil.java:737)
        at app//net.snowflake.client.core.HttpUtil.executeRequest(HttpUtil.java:672)
        at app//net.snowflake.client.core.HttpUtil.executeGeneralRequest(HttpUtil.java:594)
        at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.sendBatch(TelemetryClient.java:336)
        at app//net.snowflake.client.jdbc.telemetry.TelemetryClient.lambda$sendBatchAsync$0(TelemetryClient.java:267)
        at app//net.snowflake.client.jdbc.telemetry.TelemetryClient$$Lambda$280/0x000000080102a880.call(Unknown Source)
        at java.base@17.0.4.1/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.4.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base@17.0.4.1/java.lang.Thread.run(Thread.java:833)

Contribute

marcosmarxm commented 1 year ago

Can you share the complete log file? This is only a WARN and not an error.

sivankumar86 commented 1 year ago

d97c7aaa_ba56_4a4c_94c3_88921c0e031d_logs_19669_txt.log Attached full log

marcosmarxm commented 1 year ago

@sivankumar86 can you give more information about your setup what is the configuration of your server (memory, disk and cpu)? Are you not seen any data in your destination? Are all connections failing or you have some working?

sivankumar86 commented 1 year ago

Airbyte is running on k8s. find pod configuration and i donot think , it is related to resource issue as same config works with no issue on 0.40.18.

jobResources: requests: cpu: 1000m memory: 4096Mi limits: cpu: 2000m memory: 6240Mi

Are you not seen any data in your destination? ==> I can see the data but, i want to know whether i can ignore this warning safely

Are all connections failing or you have some working? Connections are not failing but, all the logs contains this warning message.

evantahler commented 1 year ago

Closing as this is only a warning