search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.12k stars 4.12k forks source link

Google-Ads and Bing-Ads source connectors suddenly fail to get data nor pass the connectivity test #38699

Closed betizad closed 5 months ago

betizad commented 5 months ago

Connector Name

source-google-ads and source-bing-ads

Connector Version

3.5.0 and 2.6.1

What step the error happened?

During the sync

Aitbyte version

0.61.0 Open Source Self hosted in Azure VM

Relevant information

Both connectors where functional and loading data into destination. Over the weekend, the connectors failed to load any more and I get an error while testing the connectors:

Internal message: Check failed because of an internal error
Failure origin: source

Relevant log output

io.temporal.failure.ActivityFailure: Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=a31922dc-87c2-3bfe-a607-b0274d975869, identity='1@50f70cbc555c', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
    at java.base/java.lang.Thread.getStackTrace(Thread.java:2450)
    at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49)
    at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
    at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)
    at jdk.proxy2/jdk.proxy2.$Proxy118.runWithJobOutput(Unknown Source)
    at io.airbyte.workers.temporal.check.connection.CheckConnectionWorkflowImpl.run(CheckConnectionWorkflowImpl.java:55)
    at CheckConnectionWorkflowImplProxy.run$accessor$Ut4ar92T(Unknown Source)
    at CheckConnectionWorkflowImplProxy$auxiliary$gmM4yDcD.call(Unknown Source)
    at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79)
    at CheckConnectionWorkflowImplProxy.run(Unknown Source)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339)
    at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314)
    at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70)
    at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135)
    at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102)
    at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
    at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.temporal.failure.ApplicationFailure: message='io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.', type='java.lang.RuntimeException', nonRetryable=false
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:62)
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:211)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243)
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216)
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.temporal.failure.ApplicationFailure: message='Unexpected error while getting checking connection.', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:135)
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44)
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142)
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:226)
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57)
    ... 14 more
    Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/3bceb483-2d0f-49bb-8278-10f542a78260/0', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
    at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:191)
    at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:149)
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71)
    ... 18 more
Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/3bceb483-2d0f-49bb-8278-10f542a78260/0', type='java.nio.file.NoSuchFileException', nonRetryable=false
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438)
    at java.base/java.nio.file.Files.createDirectory(Files.java:699)
    at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:121)
    ... 20 more

Contribute

betizad commented 5 months ago

I digged a bit deeper and ran airbyte in debug mode (with -x flag). Then I see that I get an strange error:

2024-05-28T13:17:28.057703554Z Activity Executor taskQueue="CHECK_CONNECTION", namespace="default": 14 ERROR FileManager (/tmp/workspace/47221804-b526-4b1c-bbd2-0fc698cfb878/0/logs.log) java.io.IOException: Could not create directory /tmp/workspace/47221804-b526-4b1c-bbd2-0fc698cfb878/0 java.io.IOException: Could not create directory /tmp/workspace/47221804-b526-4b1c-bbd2-0fc698cfb878/0

Any ideas, why is this suddenly hapenning?

BrianAndersen78 commented 5 months ago

Got the same error... I updated from 0.50 to 0.61 and all source connections start failing with this error. Creating new sources results in same error.

BrianAndersen78 commented 5 months ago

I did a docker volume rm $(docker volume ls -q | grep airbyte) and started over, and it fixed the problem.

betizad commented 5 months ago

I did use the fix as mentioned here and it worked temporarily. It would be nice that it did not happen in the first place after upgrade.

docker exec -u root -it airbyte-worker /bin/bash
chown -R airbyte:airbyte /tmp

https://github.com/airbytehq/airbyte/discussions/36984#discussioncomment-9086401

marcosmarxm commented 5 months ago

It looks the issue is related to the breaking change in the deployment. I'm closing as it isn't related to the connectors.