API endpoint `v1/connections` returns 500

tianchen98 commented 11 months ago

What method are you using to run Airbyte?

Kubernetes

Platform Version or Helm Chart Version

0.49.6

What step the error happened?

On deploy

Revelant information

the API endpoint v1/connections return 500. The original request is

curl --request GET --url 'https://api.airbyte.dev.k8s.pelotime.com/v1/connections' --header 'Authorization: Basic <Redacted>' --header 'accept: application/json' --header 'content-type: application/json'

and the response is {"type":"about:blank","status":500}. This seems to be caused by a bug in Airbyte server

Relevant log output

2023-11-15 20:10:50 INFO i.a.a.s.n.LoggingNettyChannelHandler(channelRead):37 - [GET] /v1/connections
2023-11-15 20:10:51 INFO i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/connections, operation=GET, status_code=500}
2023-11-15 20:10:51 ERROR i.m.h.s.RouteExecutor(logException):444 - Unexpected error occurred: Cannot invoke "io.airbyte.api.client.model.generated.ConnectionSchedule$TimeUnitEnum.getValue()" because the return value of "io.airbyte.api.client.model.generated.ConnectionSchedule.getTimeUnit()" is null
java.lang.NullPointerException: Cannot invoke "io.airbyte.api.client.model.generated.ConnectionSchedule$TimeUnitEnum.getValue()" because the return value of "io.airbyte.api.client.model.generated.ConnectionSchedule.getTimeUnit()" is null
    at io.airbyte.api.server.mappers.ConnectionReadMapper.from(ConnectionReadMapper.kt:68) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.mappers.ConnectionsResponseMapper.from(ConnectionsResponseMapper.kt:46) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.services.ConnectionServiceImpl.listConnectionsForWorkspaces(ConnectionService.kt:255) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.controllers.ConnectionsController.listConnections$lambda$9(ConnectionsController.kt:224) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.apiTracking.TrackingHelper.callWithTracker(TrackingHelper.kt:105) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.controllers.ConnectionsController.listConnections(ConnectionsController.kt:223) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.airbyte.api.server.controllers.$ConnectionsController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-api-server-0.50.33.jar:?]
    at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
    at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.10.1.jar:3.10.1]
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.10.1.jar:3.10.1]
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.10.1.jar:3.10.1]
    at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.10.1.jar:3.10.1]
    at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.10.1.jar:3.10.1]
    at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.5.jar:3.5.5]
    at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.5.jar:3.5.5]
    at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.10.1.jar:3.10.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-15 20:10:51 INFO i.a.a.s.n.LoggingNettyChannelHandler(capture):87 - Request: [c054d1f8-4698-4ee5-8508-65728a415c5d] -- {"headers":{"x-datadog-trace-id":["9559351412467960"],"x-datadog-parent-id":["122092677146938934"],"x-datadog-sampling-priority":["0"],"host":["api.airbyte.dev.k8s.pelotime.com"],"x-request-id":["31f88537263b37cd9e7d334bb6e0d5b2"],"x-real-ip":["69.164.128.222"],"x-forwarded-for":["69.164.128.222"],"x-forwarded-host":["api.airbyte.dev.k8s.pelotime.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-forwarded-scheme":["https"],"x-scheme":["https"],"user-agent":["curl\/8.1.2"],"accept":["application\/json"],"content-type":["application\/json"],"x-airbyte-request-id":["c054d1f8-4698-4ee5-8508-65728a415c5d"]},"requestURI":"\/v1\/connections","bodyText":"","queryString":"","method":"GET"}
2023-11-15 20:10:51 INFO i.a.a.s.n.LoggingNettyChannelHandler(capture):88 - Response: [c054d1f8-4698-4ee5-8508-65728a415c5d] -- {"headers":{"content-type":["application\/problem+json"],"date":["Wed, 15 Nov 2023 20:10:51 GMT"],"content-length":["35"],"connection":["keep-alive"]},"status":500,"bodyText":"{\"type\":\"about:blank\",\"status\":500}"}
2023-11-15 20:10:51 INFO i.m.h.s.n.h.a.e.AccessLog(log):125 - ip-10-14-37-236.ec2.internal - - [15/Nov/2023:20:10:50 +0000] "GET /v1/connections HTTP/1.1" 500 35

juanfbl9307 commented 10 months ago

Hi, had the same issue and solve it replacing a value in the ConfigMap airbyte-airbyte-env, there is no option to do this by the values files because the helm template originally doesn't add the https:// on the value.

kind: ConfigMap name: airbyte-airbyte-env

- INTERNAL_API_HOST: airbyte-airbyte-server-svc:8001
+ INTERNAL_API_HOST: http://airbyte-airbyte-server-svc:8001

After this you must restart only the airbyte-api-server deployment.

The issue

api-server can't communicate with server because it needs the https:// on the INTERNAL_API_HOST , replacing this by hand works but then you need to redeploy. Other services as worker will fail if the get INTERNAL_API_HOST with http:// .

Chart: airbyte TEMPLATE: env-configmap.yaml

27 INTERNAL_API_HOST: {{ .Release.Name }}-airbyte-server-svc:{{ .Values.server.service.port }}

Chart: airbyte-api-server Template: deployment.yaml

58 - name: INTERNAL_API_HOST
59     valueFrom:
60         configMapKeyRef:
61              name: {{ .Release.Name}}-airbyte-env
62              key: INTERNAL_API_HOST

pdemagny commented 10 months ago

Hi, we are also impacted by this. The worker is not the only component that crashes when updating the global configmaps as shown by @juanfbl9307 , the webapp crashes too ...

The only workaround we found to work was to force the value of INTERNAL_API_HOST at the airbyte-api-server level, so in the values files this gives us:

airbyte-api-server:
  env_vars:
    INTERNAL_API_HOST: http://airbyte-airbyte-server-svc:8001

This is hardcoded and far from elementary robustness :( A cleaner solution, such as standardizing the use of this environment variables across all components, or adjusting the helm charts to take this into consideration, is still required.

juanfbl9307 commented 10 months ago

airbyte-api-server depends on {{ .Release.Name }}-airbyte-env configMap, three invironment variables from Deployment are there: airbyte-api-server-deployment

Following you approach @pdemagny, Creating a configMap only for airbyte-api-server and changing the reference on the deployment, this new configMap will have the protocol http:// before the enviornment variable definition.

[ ] Create configMap manifest on airbyte-api-server helm template path airbyte-api-server/templates/config-map.yaml with

apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-airbyte-api-server-env
data:
INTERNAL_API_HOST: http://{{ .Release.Name }}-airbyte-server-svc:{{ .Values.server.service.port }}
AIRBYTE_VERSION: {{ .Values.version | default .Chart.AppVersion }}
AIRBYTE_API_HOST: {{ .Release.Name }}-airbyte-api-server-svc:{{ index .Values "airbyte-api-server" "service" "port" }}

[ ] Replace the values reference on deployment
configMapKeyRef:
name: {{ .Release.Name }}-airbyte-env
configMapKeyRef:
name: {{ .Release.Name }}-airbyte-api-server-env

pdemagny commented 10 months ago

Yep, that's the way to go if you are willing to fork or download the chart and install it from local files while waiting for a more permanent fix.
I don't want to do that because I want to keep using the official chart, not the local one, to benefit from other updates of the chart and have my Renovate Bot scan my helmfile.yaml and keep it up to date as well, hence my dirty workaround ;)

airbytehq / airbyte