search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.17k stars 4.13k forks source link

[helm] Deployment with external minio: ` Cannot create enum from ${env:S3_LOG_BUCKET_REGION} ` #34539

Open leonardo-lages-hyperflow opened 9 months ago

leonardo-lages-hyperflow commented 9 months ago

Platform Version

0.50.14

What step the error happened?

On deploy

Revelant information

When the worker start happens an error which blocks the use of connectors.

The documentation tells to disable minio, enable externalMinio and then put the s3 region empty. But when we do this occurs the error showed in the log.

Relevant log output

Unsetting empty environment variable 'ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS'
Unsetting empty environment variable 'GOOGLE_APPLICATION_CREDENTIALS'
Unsetting empty environment variable 'S3_LOG_BUCKET_REGION'
Unsetting empty environment variable 'OTEL_COLLECTOR_ENDPOINT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_CPU_REQUEST'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_MEMORY_REQUEST'
Unsetting empty environment variable 'ACTIVITY_MAX_ATTEMPT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_CPU_LIMIT'
Unsetting empty environment variable 'VERSION'
Unsetting empty environment variable 'WORKFLOW_FAILURE_RESTART_DELAY_SECONDS'
Unsetting empty environment variable 'METRIC_CLIENT'
Unsetting empty environment variable 'JOB_MAIN_CONTAINER_MEMORY_LIMIT'
Unsetting empty environment variable 'GCS_LOG_BUCKET'
Unsetting empty environment variable 'ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS'
INFO StatusConsoleListener Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.10.1)

2024-01-25 19:26:22 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [k8s, cloud, ec2, control-plane]
2024-01-25 19:26:23 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):111 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2024-01-25 19:26:23 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable METRIC_CLIENT: ''
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable METRIC_CLIENT: ''
2024-01-25 19:26:26 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-25 19:26:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-25 19:26:26 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):148 - Waiting for temporal server...
2024-01-25 19:26:26 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):159 - Waiting for namespace default to be initialized in temporal...
2024-01-25 19:26:28 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
2024-01-25 19:26:34 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):176 - Temporal namespace default initialized!
2024-01-25 19:26:34 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):167 - Initializing common worker dependencies.
2024-01-25 19:26:34 INFO i.a.w.ApplicationInitializer(configureTemporal):322 - Configuring Temporal....
ERROR StatusConsoleListener Could not create plugin of type class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.RuntimeException: Cannot build appender due to errors
 java.lang.RuntimeException: Cannot build appender due to errors
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:147)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:25)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
    at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
    at io.airbyte.workers.ApplicationInitializer.configureTemporal(ApplicationInitializer.java:322)
    at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:177)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:143)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:57)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60)
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229)
    at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:600)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:594)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104)
    at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:79)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:323)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:309)
    at io.airbyte.workers.Application.main(Application.java:15)
Caused by: java.lang.IllegalArgumentException: Cannot create enum from ${env:S3_LOG_BUCKET_REGION} value!
    at com.amazonaws.regions.Regions.fromName(Regions.java:105)
    at com.van.logging.aws.S3Configuration.resolveRegion(S3Configuration.java:204)
    at com.van.logging.aws.S3Configuration.setRegion(S3Configuration.java:125)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.getS3ConfigIfEnabled(Log4j2AppenderBuilder.java:172)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:271)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:140)
    ... 46 more
ERROR StatusConsoleListener Unable to invoke factory method in class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender
 java.lang.IllegalStateException: No factory method found for class com.van.logging.log4j2.Log4j2Appender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:268)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:140)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
    at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
    at io.airbyte.workers.ApplicationInitializer.configureTemporal(ApplicationInitializer.java:322)
    at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:177)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:143)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:57)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60)
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229)
    at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:600)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:594)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104)
    at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:79)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:323)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:309)
    at io.airbyte.workers.Application.main(Application.java:15)
ERROR StatusConsoleListener Unable to create Appender of type Log4j2Appender
2024-01-25 19:26:34 INFO i.a.c.t.TemporalUtils(configureTemporalNamespace):123 - Workflow execution TTL already set for namespace default. Remains unchanged as: 30 days
ERROR StatusConsoleListener Could not create plugin of type class com.van.logging.log4j2.Log4j2Appender for element Log4j2Appender: java.lang.RuntimeException: Cannot build appender due to errors
 java.lang.RuntimeException: Cannot build appender due to errors
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:147)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:25)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1162)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1083)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:323)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:295)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:253)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.appender.rewrite.RewriteAppender.append(RewriteAppender.java:89)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
    at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2933)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2886)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2868)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2652)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2400)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
    at io.airbyte.commons.temporal.TemporalUtils.configureTemporalNamespace(TemporalUtils.java:123)
    at io.airbyte.workers.ApplicationInitializer.configureTemporal(ApplicationInitializer.java:324)
    at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:177)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:143)
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:57)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262)
    at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60)
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229)
    at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:600)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:594)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298)
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104)
    at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81)
    at java.base/java.util.Optional.ifPresent(Optional.java:178)
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:79)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:323)
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:309)
    at io.airbyte.workers.Application.main(Application.java:15)
Caused by: java.lang.IllegalArgumentException: Cannot create enum from ${env:S3_LOG_BUCKET_REGION} value!
    at com.amazonaws.regions.Regions.fromName(Regions.java:105)
    at com.van.logging.aws.S3Configuration.resolveRegion(S3Configuration.java:204)
    at com.van.logging.aws.S3Configuration.setRegion(S3Configuration.java:125)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.getS3ConfigIfEnabled(Log4j2AppenderBuilder.java:172)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:271)
    at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:140)
leonardo-lages-hyperflow commented 9 months ago

If we put the region occurs another error that showed in the log below:

Caused by: java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set. at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:450) at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) at com.van.logging.aws.AwsClientHelpers.buildClient(AwsClientHelpers.java:88) at com.van.logging.aws.S3PublishHelper.(S3PublishHelper.java:52) at com.van.logging.log4j2.Log4j2AppenderBuilder.lambda$createCachePublisher$0(Log4j2AppenderBuilder.java:276) at java.base/java.util.Optional.ifPresent(Optional.java:178) at com.van.logging.log4j2.Log4j2AppenderBuilder.createCachePublisher(Log4j2AppenderBuilder.java:271) at com.van.logging.log4j2.Log4j2AppenderBuilder.build(Log4j2AppenderBuilder.java:140) ... 52 more

marcosmarxm commented 9 months ago

@jdpgrailsdev is this the problem you had with the latest version of Airbyte and Log4j?

marcosmarxm commented 9 months ago

I think the solution was done by https://github.com/airbytehq/airbyte-platform/commit/28e4bfc35b31a2535c0970dfcfd77fcf72b53ca4 probably need wait a new release of Helm Chart.

marcosmarxm commented 9 months ago

@leonardo-lages-hyperflow can you share the Chart App version you're using?

marcosmarxm commented 9 months ago

@leonardo-lages-hyperflow there is a new release platform 0.50.45 and chart 0.50.115 please run some tests with the new version

marcosmarxm commented 9 months ago

Confirmed the logs are showing in latest Helm version

image
leonardo-lages-hyperflow commented 9 months ago

Hi @marcosmarxm that problem was solved Thank you!

But now I'm getting another problem, the worker can't connect to the external minio, as you can see in the log bellow:

2024-01-27 15:56:38 INFO i.a.w.ApplicationInitializer(registerWorkerFactory):182 - Registering worker factories.... 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerGetSpec):278 - Get Spec Workflow registered. 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerCheckConnection):231 - Check Connection Workflow registered. 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerDiscover):264 - Discover Workflow registered. 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerSync):291 - Registering sync workflow for task queue 'SYNC'... 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerSync):301 - Sync Workflow registered. 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerConnectionManager):247 - Connection Manager Workflow registered. 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(registerNotification):209 - registering new notification workflow 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(onApplicationEvent):149 - Starting worker factory... 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="DISCOVER_SCHEMA", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="DISCOVER_SCHEMA", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="NOTIFY", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="NOTIFY", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-worker-69cbb669fc-wdsq2} 2024-01-27 15:56:39 INFO i.a.w.ApplicationInitializer(onApplicationEvent):152 - Application initialized. 2024-01-27 15:56:39 appender-core-5.0.0.jar:?:98 - Startup completed in 17084ms. Server Running: http://airbyte-worker-69cbb669fc-wdsq2:9000 2024-01-27 15:57:33 ERROR c.v.l.BufferPublisher(startPublish):38 - Cannot start publish with com.van.logging.aws.S3PublishHelper@fd56c64 due to error java.lang.RuntimeException: Cannot start publishing: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null) at com.van.logging.aws.S3PublishHelper.start(S3PublishHelper.java:80) ~[appender-core-5.0.0.jar:?] at com.van.logging.BufferPublisher.startPublish(BufferPublisher.java:36) ~[appender-core-5.0.0.jar:?] at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:199) ~[appender-core-5.0.0.jar:?] at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:255) ~[appender-core-5.0.0.jar:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1445) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1381) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.van.logging.aws.S3PublishHelper.start(S3PublishHelper.java:73) ~[appender-core-5.0.0.jar:?] ... 8 more 2024-01-27 15:57:33 ERROR c.v.l.BufferPublisher(endPublish):69 - Cannot end publish with com.van.logging.aws.S3PublishHelper@fd56c64 due to error java.lang.RuntimeException: Cannot end publishing: Cannot publish to S3: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null) at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:66) ~[appender-core-5.0.0.jar:?] at com.van.logging.BufferPublisher.endPublish(BufferPublisher.java:67) ~[appender-core-5.0.0.jar:?] at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:210) ~[appender-core-5.0.0.jar:?] at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:255) ~[appender-core-5.0.0.jar:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Caused by: java.lang.RuntimeException: Cannot publish to S3: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null) at com.van.logging.aws.S3PublishHelper.publishFile(S3PublishHelper.java:131) ~[appender-core-5.0.0.jar:?] at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:61) ~[appender-core-5.0.0.jar:?] ... 8 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.510.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816) ~[aws-java-sdk-s3-1.12.6.jar:?] at com.van.logging.aws.S3PublishHelper.publishFile(S3PublishHelper.java:126) ~[appender-core-5.0.0.jar:?] at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:61) ~[appender-core-5.0.0.jar:?] ... 8 more

marcosmarxm commented 9 months ago

I recommend to use the default minio for now. There are some work being done to improve the existing chart.

raphaelauv commented 9 months ago

I make it workairbyte/worker:0.50.47 and helm chart 0.53.52

-> https://github.com/airbytehq/airbyte/issues/31988#issuecomment-1927252063