search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.88k stars 3.82k forks source link

[metric-reporter] OTEL collector fails with Invalid endpoint, must start with http:// or https://: #36045

Open dmvianna opened 4 months ago

dmvianna commented 4 months ago

Platform Version

0.52.1

What step the error happened?

None

Revelant information

With .env, docker-compose.yaml, flags.yml and temporal/dynamicconfig/development.yaml in the same directory, I can execute

set -a
source .env
docker compose up

And Airbyte works as expected. Now if I make the following changes, as per the documentation:

.env

# Open Telemetry
PUBLISH_METRICS=true
METRIC_CLIENT=otel
OTEL_COLLECTOR_ENDPOINT=http://otel-collector:4317

docker-compose.yaml

  metric-reporter:
    image: airbyte/metrics-reporter:${VERSION}
    container_name: metric-reporter
    networks:
      - airbyte_internal
    environment:
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - DATABASE_URL=${DATABASE_URL}
      - DATABASE_USER=${DATABASE_USER}
      - METRIC_CLIENT=${METRIC_CLIENT}
      - OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}
  otel-collector:
    image: otel/opentelemetry-collector-contrib
    volumes:
      - ./otel-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension

And create a new file otel-config.yaml

extensions:
  zpages:
    endpoint: localhost:55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 2G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

  extensions: [zpages]

then airbyte-cron throws an error, and airbyte gets into an unhealthy state.

Relevant log output

airbyte-api-server                | 
airbyte-api-server                |     ___    _      __          __
airbyte-api-server                |    /   |  (_)____/ /_  __  __/ /____
airbyte-api-server                |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-api-server                |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-api-server                | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-api-server                |                     /____/
airbyte-api-server                |         : airbyte-api-server :
airbyte-api-server                | --------------------------------------------------
airbyte-api-server                |  Will soon be available at http://localhost:8006/
airbyte-api-server                | --------------------------------------------------
airbyte-api-server                | 
airbyte-cron                      | 
airbyte-cron                      |     ___    _      __          __
airbyte-cron                      |    /   |  (_)____/ /_  __  __/ /____
airbyte-cron                      |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-cron                      |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-cron                      | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-cron                      |                     /____/
airbyte-cron                      |         : airbyte-cron :
airbyte-cron                      | 
airbyte-worker                    | 
airbyte-worker                    |     ___    _      __          __
airbyte-worker                    |    /   |  (_)____/ /_  __  __/ /____
airbyte-worker                    |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-worker                    |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-worker                    | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-worker                    |                     /____/
airbyte-worker                    |         : airbyte-workers :
airbyte-worker                    | 
airbyte-server                    | 
airbyte-server                    |     ___    _      __          __
airbyte-server                    |    /   |  (_)____/ /_  __  __/ /____
airbyte-server                    |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-server                    |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-server                    | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-server                    |                     /____/
airbyte-server                    |         : airbyte-server :
airbyte-server                    | --------------------------------------------------
airbyte-server                    |  Will soon be available at http://localhost:8000/
airbyte-server                    | --------------------------------------------------
airbyte-server                    | 
airbyte-connector-builder-server  | 
airbyte-connector-builder-server  |     ___    _      __          __
airbyte-connector-builder-server  |    /   |  (_)____/ /_  __  __/ /____
airbyte-connector-builder-server  |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-connector-builder-server  |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-connector-builder-server  | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-connector-builder-server  |                     /____/
airbyte-connector-builder-server  |         : airbyte-connector-builder-server :
airbyte-connector-builder-server  | 
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.227Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.24.0.2:7233"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.24.0.2:7234"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.24.0.2:7239"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.229Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.24.0.2:7234"],"logging-call-at":"service_resolver.go:279"}
airbyte-temporal                  | {"level":"info","ts":"2024-03-14T04:24:50.230Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.24.0.2:7235"],"logging-call-at":"service_resolver.go:279"}
airbyte-cron                      | 2024-03-14 04:24:50 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-worker                    | 2024-03-14 04:24:51 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-server                    | 2024-03-14 04:24:51 INFO i.m.c.e.DefaultEnvironment(<init>):168 - Established active environments: [control-plane]
airbyte-cron                      | 2024-03-14 04:24:51 INFO i.a.m.l.MetricClientFactory(initializeOpenTelemetryMetricClient):131 - Initializing OpenTelemetryMetricClient
airbyte-cron                      | 2024-03-14 04:24:51 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):147 - Waiting for temporal server...
airbyte-cron                      | 2024-03-14 04:24:51 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):158 - Waiting for namespace default to be initialized in temporal...
airbyte-cron                      | 2024-03-14 04:24:51 ERROR i.m.s.TaskExceptionHandler(handleCreationFailure):44 - Error creating scheduled task for bean [WorkspaceCleaner] Error instantiating bean of type  [io.airbyte.cron.jobs.WorkspaceCleaner]
airbyte-cron                      | 
airbyte-cron                      | Message: Invalid endpoint, must start with http:// or https://: 
airbyte-cron                      | Path Taken: new WorkspaceCleaner(MetricClient metricClient) --> new WorkspaceCleaner([MetricClient metricClient])
airbyte-cron                      | io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.cron.jobs.WorkspaceCleaner]
airbyte-cron                      | 
airbyte-cron                      | Message: Invalid endpoint, must start with http:// or https://: 
airbyte-cron                      | Path Taken: new WorkspaceCleaner(MetricClient metricClient) --> new WorkspaceCleaner([MetricClient metricClient])
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2347) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2302) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2314) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3116) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3018) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2979) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2753) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1752) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:89) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2165) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1328) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.airbyte.cron.jobs.$WorkspaceCleaner$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-cron-0.52.1.jar:?]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2332) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2302) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2314) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3116) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3018) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2979) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2753) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1752) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:858) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     at io.micronaut.scheduling.processor.ScheduledMethodProcessor.lambda$process$2(ScheduledMethodProcessor.java:123) ~[micronaut-context-4.3.10.jar:4.3.10]
airbyte-cron                      |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
airbyte-cron                      |     at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
airbyte-cron                      |     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
airbyte-cron                      |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
airbyte-cron                      |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
airbyte-cron                      |     at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
airbyte-cron                      | Caused by: java.lang.IllegalArgumentException: Invalid endpoint, must start with http:// or https://: 
airbyte-cron                      |     at io.opentelemetry.exporter.internal.ExporterBuilderUtil.validateEndpoint(ExporterBuilderUtil.java:30) ~[opentelemetry-exporter-common-1.36.0.jar:1.36.0]
airbyte-cron                      |     at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.setEndpoint(GrpcExporterBuilder.java:104) ~[opentelemetry-exporter-common-1.36.0.jar:1.36.0]
airbyte-cron                      |     at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.setEndpoint(OtlpGrpcSpanExporterBuilder.java:130) ~[opentelemetry-exporter-otlp-1.36.0.jar:1.36.0]
airbyte-cron                      |     at io.airbyte.metrics.lib.OpenTelemetryMetricClient.initialize(OpenTelemetryMetricClient.java:120) ~[io.airbyte.airbyte-metrics-metrics-lib-0.52.1.jar:?]
airbyte-cron                      |     at io.airbyte.metrics.lib.MetricClientFactory.initializeOpenTelemetryMetricClient(MetricClientFactory.java:133) ~[io.airbyte.airbyte-metrics-metrics-lib-0.52.1.jar:?]
airbyte-cron                      |     at io.airbyte.metrics.lib.MetricClientFactory.initialize(MetricClientFactory.java:69) ~[io.airbyte.airbyte-metrics-metrics-lib-0.52.1.jar:?]
airbyte-cron                      |     at io.airbyte.cron.config.ApplicationBeanFactory.metricClient(ApplicationBeanFactory.java:32) ~[io.airbyte-airbyte-cron-0.52.1.jar:?]
airbyte-cron                      |     at io.airbyte.cron.config.$ApplicationBeanFactory$MetricClient0$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-cron-0.52.1.jar:?]
airbyte-cron                      |     at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2332) ~[micronaut-inject-4.3.10.jar:4.3.10]
airbyte-cron                      |     ... 29 more
airbyte-connector-builder-server  | 2024-03-14 04:24:51 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
airbyte-worker                    | 2024-03-14 04:24:51 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
airbyte-api-server                | 2024-03-14 04:24:52 INFO i.m.r.Micronaut(start):100 - Startup completed in 4070ms. Server Running: http://29ddf4998ccc:8006
airbyte-server                    | 2024-03-14 04:24:53 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):147 - Waiting for temporal server...
airbyte-server                    | 2024-03-14 04:24:53 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):158 - Waiting for namespace default to be initialized in temporal...
airbyte-cron                      | 2024-03-14 04:24:54 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
airbyte-connector-builder-server  | 2024-03-14 04:24:54 INFO i.m.r.Micronaut(start):100 - Startup completed in 4810ms. Server Running: http://2effc7ac94fd:80
airbyte-worker                    | 2024-03-14 04:24:54 INFO i.a.m.l.MetricClientFactory(initializeOpenTelemetryMetricClient):131 - Initializing OpenTelemetryMetricClient
airbyte-worker                    | 2024-03-14 04:24:54 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):147 - Waiting for temporal server...
airbyte-worker                    | 2024-03-14 04:24:54 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):158 - Waiting for namespace default to be initialized in temporal...
airbyte-server                    | 2024-03-14 04:24:56 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
airbyte-worker                    | 2024-03-14 04:24:57 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
airbyte-cron                      | 2024-03-14 04:24:59 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):175 - Temporal namespace default initialized!
airbyte-cron                      | 2024-03-14 04:24:59 INFO i.a.m.l.MetricClientFactory(initializeOpenTelemetryMetricClient):131 - Initializing OpenTelemetryMetricClient
airbyte-cron                      | 2024-03-14 04:24:59 ERROR i.m.s.TaskExceptionHandler(handleCreationFailure):44 - Error creating scheduled task for bean [SelfHealTemporalWorkflows] Error instantiating bean of type  [io.airbyte.commons.temporal.WorkflowClientWrapped]
ggam commented 2 months ago

Happens to me too

tom-bazin commented 1 month ago

Hi @dmvianna, @ggam,

Please add missing environnement variable OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT} to services server, airbyte-cron, and airbyte-connector-builder-server.

It worked fine for me with these parameters.
I still need to dive into the code to understand the purpose of metric-reporter as it seems the other services talk directly to OpenTelemetry collector.