airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.45k stars 3.98k forks source link

[source-Mysql] `JAVA_HOME` is not correct #32435

Closed minguyen9988 closed 3 months ago

minguyen9988 commented 10 months ago

Connector Name

source-mysql

Connector Version

airbyte/source-mysql:3.1.7

What step the error happened?

Configuring a new connector

Relevant information

I clone and run with docker-compose as per instruction. But this mysql source image return this.

2023-11-10 17:27:38 INFO i.a.w.p.DockerProcessFactory(create):187 - Preparing command: docker run --rm --init -i -w /data/2d983f87-d067-4b8d-a20a-e37febed805c/0 --log-driver none --name source-mysql-check-2d983f87-d067-4b8d-a20a-e37febed805c-0-rmjfm --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-mysql:3.1.7 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.34 -e WORKER_JOB_ID=2d983f87-d067-4b8d-a20a-e37febed805c airbyte/source-mysql:3.1.7 check --config source_config.json 2023-11-10 17:27:38 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):186 - Reading messages from protocol version 0.2.0 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - ERROR: JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-17-amazon-corretto 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - Please set the JAVA_HOME variable in your environment to match the 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - location of your Java installation. 2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 2023-11-10 17:27:39 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):133 - Unexpected error while checking connection: io.airbyte.workers.exception.WorkerException: Error checking connection status: no status nor failure reason were outputted at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:268) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?] at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]

at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?] at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?] at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?] at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?] at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:118) ~[io.airbyte-airbyte-workers-0.50.34.jar:?] at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Relevant log output

2023-11-10 17:27:38 INFO i.a.w.p.DockerProcessFactory(create):187 - Preparing command: docker run --rm --init -i -w /data/2d983f87-d067-4b8d-a20a-e37febed805c/0 --log-driver none --name source-mysql-check-2d983f87-d067-4b8d-a20a-e37febed805c-0-rmjfm --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-mysql:3.1.7 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.34 -e WORKER_JOB_ID=2d983f87-d067-4b8d-a20a-e37febed805c airbyte/source-mysql:3.1.7 check --config source_config.json
2023-11-10 17:27:38 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):186 - Reading messages from protocol version 0.2.0
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - ERROR: JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-17-amazon-corretto
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - Please set the JAVA_HOME variable in your environment to match the
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - location of your Java installation.
2023-11-10 17:27:38 ERROR i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-10 17:27:39 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):133 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Error checking connection status: no status nor failure reason were outputted
    at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:268) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:118) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Contribute

minguyen9988 commented 10 months ago

https://unix.stackexchange.com/questions/671246/why-is-bash-not-evaluating-the-executable-bit-correctly-in-alpine-3-14-2 This is the problem, the container has problem.


bash-5.2# ^C
bash-5.2# if [ ! -x /usr/lib/jvm/java-17-amazon-corretto/bin/java ] ; then         echo "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.";     fi
ERROR: JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-17-amazon-corretto

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.```
evantahler commented 4 months ago

Since this issue was written, we've dramatically revamped the connector. Please try your sync again with the latest connectors and platform!

asif-srti commented 3 months ago

hi @evantahler release v0.59.1 also has this issue and it seems to be generic across all SQL destinations.

evantahler commented 3 months ago

Can you share as much information as possible about how you are running airbyte? We don't see this issue in any of our test environments:

asif-srti commented 3 months ago

Hi

SHELL=/bin/bash
TERM=xterm-256color
USER=root
LS_COLORS=xxx
SUDO_USER=ubuntu
SUDO_UID=1000
KUBECONFIG=xxx
USERNAME=root
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PWD=/home/ubuntu
LANG=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/su
HOME=/root
LOGNAME=root
LESSOPEN=| /usr/bin/lesspipe %s
SUDO_GID=1000
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env`
evantahler commented 3 months ago

Thanks for the info! In the interum, would you mind trying out abctl, the now recommended way to run airbyte?

evantahler commented 3 months ago

Closing due to lack of response, and inability to reproduce.