search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.72k stars 3.78k forks source link

[helm] Cannot run more than two worker #38873

Open firehist opened 3 weeks ago

firehist commented 3 weeks ago

Helm Chart Version

0.101.2

What step the error happened?

Other

Relevant information

We are running Airbyte, thanks to the official helm chart in version 0.101.2.

We are using external Google Secret Key Manager, Google Cloud Storage and our PostgreSQL on Google Cloud SQL as external in our values.yaml.

Everything is working well, except when I have the great idea to scale up worker > 1.

Other workers will fail with the attached log ouput.

It seems to receive a string //airbyte-airbyte-server-svc" which should be the internal API HOST or something like that, but the parsePort() fails, as there is no port in this string... The second worker POD definition is exactly the same than for the first one, as it is controlled by a Deployment.

Relevant log output

at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2732) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1794) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:112) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2305) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForField(AbstractInitializableBeanDefinition.java:1904) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.airbyte.workers.$ApplicationInitializer$Definition.inject(Unknown Source) ~[io.airbyte-airbyte-workers-0.61.0.jar:?]
    at io.airbyte.workers.$ApplicationInitializer$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-workers-0.61.0.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2311) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2281) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2293) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3095) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2997) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2958) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2932) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3521) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3476) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3450) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1432) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:878) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:868) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.lambda$$1(ApplicationEventPublisherFactory.java:215) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.core.util.SupplierUtil$1.get(SupplierUtil.java:47) ~[micronaut-core-4.4.8.jar:4.4.8]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery.onStart(NettyServiceDiscovery.java:54) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.discovery.$NettyServiceDiscovery$Definition$Exec.dispatch(Unknown Source) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:3856) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.aop.chain.AdapterIntroduction.intercept(AdapterIntroduction.java:84) ~[micronaut-aop-4.4.8.jar:4.4.8]
    at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:138) ~[micronaut-aop-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery$ApplicationEventListener$onStart1$Intercepted.onApplicationEvent(Unknown Source) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226) ~[micronaut-inject-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:679) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:317) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:110) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:80) [micronaut-context-4.4.8.jar:4.4.8]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:328) [micronaut-context-4.4.8.jar:4.4.8]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:314) [micronaut-context-4.4.8.jar:4.4.8]
    at io.airbyte.workers.Application.main(Application.java:15) [io.airbyte-airbyte-workers-0.61.0.jar:?]
Caused by: java.lang.NumberFormatException: For input string: "//airbyte-airbyte-server-svc"
    at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
    at java.base/java.lang.Integer.parseInt(Integer.java:648) ~[?:?]
    at java.base/java.lang.Integer.parseInt(Integer.java:778) ~[?:?]
    at io.airbyte.workers.config.ApiClientBeanFactory.parsePort(ApiClientBeanFactory.java:160) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
    at io.airbyte.workers.config.ApiClientBeanFactory.apiClient(ApiClientBeanFactory.java:53) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
    at io.airbyte.workers.config.$ApiClientBeanFactory$ApiClient0$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2311) ~[micronaut-inject-4.4.8.jar:4.4.8]
    ... 85 more
firehist commented 3 weeks ago

Waouh, problem fixed by removing the http:// prefix in the INTERNAL_API_HOST environment variable (coming from airbyte-airbyte-env config map).

It can be related to the latest chart release ? https://artifacthub.io/packages/helm/airbyte/airbyte/0.101.2?modal=template&template=airbyte-db.yaml&compare-to=0.94.1

marcosmarxm commented 3 weeks ago

Thanks for reporting the issue @firehist I added this to deployment team backlog for further investigation