search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.76k stars 4.04k forks source link

airbyte-worker throws error while trying to initialize log4j appender when S3 log storage is selected in helm #30189

Open anjshrg opened 1 year ago

anjshrg commented 1 year ago

Topic

S3 Log Appender is not initialized

Revelant information

airbyte worker is not able to initialize a stream when log is externalized to S3. We get the following error:

Registering AWS S3 publish helper -> S3 configuration (aws-logs-594542361424-ap-south-1:job-logging/workspace/291764/0/logs.log in region ap-south-1; compressed: false)
2023-09-06 07:25:33,736 pool-523425-thread-2 ERROR Could not create plugin of type class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.RuntimeException: Cannot build appender due to errors java.lang.RuntimeException: Cannot build appender due to errors
at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:144)
at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:25)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:310)
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:282)
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:240)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:84)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2040)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1907)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:190)
at io.airbyte.workers.internal.VersionedAirbyteStreamFactory.create(VersionedAirbyteStreamFactory.java:177)
at io.airbyte.workers.internal.DefaultAirbyteSource.start(DefaultAirbyteSource.java:94)
at io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.java:137)
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$1(BufferedReplicationWorker.java:152)
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:203)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.
at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:450)
at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at com.van.logging.aws.AwsClientHelpers.buildClient(AwsClientHelpers.java:88)
at com.van.logging.aws.S3PublishHelper.<init>(S3PublishHelper.java:53)
at com.van.logging.log4j2.Log4j2AppenderBuilder.lambda$createCachePublisher$0(Log4j2AppenderBuilder.java:271)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:266)
at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:137)
... 37 more
2023-09-05 17:55:03,188 Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 196 ERROR Unable to invoke factory method in class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:260)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:310)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:282)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:240)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
    at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:84)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
    at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2028)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1891)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:195)
    at io.airbyte.commons.temporal.StreamResetRecordsHelper.deleteStreamResetRecordsForJob(StreamResetRecordsHelper.java:50)
    at io.airbyte.workers.temporal.scheduling.activities.StreamResetActivityImpl.deleteStreamResetRecordsForJob(StreamResetActivityImpl.java:38)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95)
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179)
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1589)

Because of this logs are not visible on UI for any connection - it shows a fixed log of un-related connection.

anjshrg commented 1 year ago

Looks like this is a known issue as highlighted in issues https://github.com/airbytehq/airbyte/issues/29167 and https://github.com/airbytehq/airbyte/issues/29173

samcole-p1 commented 6 months ago

This occurs for us on 0.50.54

octavia-squidington-iii commented 1 day ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.