search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

Platform Issue: Normalization docker image is not running on any syncs #25799

Closed emilmar closed 1 year ago

emilmar commented 1 year ago

Platform Version

0.44.4

What step the error happened?

During the Sync

Revelant information

Is this your first time deploying Airbyte?: No OS Version / Instance: OS image: debian-11-bullseye-v20221206 Deployment: BigQuery Compute Engine

I did a massive upgrade this morning and I must have missed something critical. Whenever I run a sync from both airbyte/source-zendesk-support and airbyte/source-salesforce to airbyte/destination-bigquerythe sync of records works fine, but the normalization docker image does not run. After the first run, on which I get an error, I no longer receive any errors and the sync finishes as "successful" despite never normalizing the data.

What I attempted this morning was:

An example of a sync failing:

Since the issue happens with two different source connectors and the log message I receive is related to normalization, I do not think it has anything to do with the sources. I think something related to my docker containers is the fault. I am willing to provide as much information as requested, I just don't even know where to begin to look. 🤗

Relevant log output

2023-05-04 12:11:15 INFO i.a.w.g.DefaultReplicationWorker(getReplicationOutput):569 - sync summary: {
  "status" : "completed",
  "recordsSynced" : 2336,
  "bytesSynced" : 6955374,
  "startTime" : 1683202234047,
  "endTime" : 1683202275159,
  "totalStats" : {
    "bytesCommitted" : 6955374,
    "bytesEmitted" : 6955374,
    "destinationStateMessagesEmitted" : 1,
    "destinationWriteEndTime" : 1683202275158,
    "destinationWriteStartTime" : 1683202234191,
    "meanSecondsBeforeSourceStateMessageEmitted" : 23,
    "maxSecondsBeforeSourceStateMessageEmitted" : 23,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 4,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 4,
    "recordsEmitted" : 2336,
    "recordsCommitted" : 2336,
    "replicationEndTime" : 1683202275159,
    "replicationStartTime" : 1683202234047,
    "sourceReadEndTime" : 1683202270680,
    "sourceReadStartTime" : 1683202234099,
    "sourceStateMessagesEmitted" : 1
  },
  "streamStats" : [ {
    "streamName" : "ticket_comments",
    "streamNamespace" : "zendesk",
    "stats" : {
      "bytesCommitted" : 6955374,
      "bytesEmitted" : 6955374,
      "recordsEmitted" : 2336,
      "recordsCommitted" : 2336
    }
  } ]
}
2023-05-04 12:11:15 INFO i.a.w.g.DefaultReplicationWorker(getReplicationOutput):570 - failures: [ ]
2023-05-04 12:11:15 INFO i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-05-04 12:11:15 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-05-04 12:11:15 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END REPLICATION -----
2023-05-04 12:11:15 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-05-04 12:11:15 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):157 - sync summary: io.airbyte.config.StandardSyncOutput@5c938e7a[standardSyncSummary=io.airbyte.config.StandardSyncSummary@1ae8e86[status=completed,recordsSynced=2336,bytesSynced=6955374,startTime=1683202234047,endTime=1683202275159,totalStats=io.airbyte.config.SyncStats@1e06f181[bytesCommitted=6955374,bytesEmitted=6955374,destinationStateMessagesEmitted=1,destinationWriteEndTime=1683202275158,destinationWriteStartTime=1683202234191,estimatedBytes=<null>,estimatedRecords=<null>,meanSecondsBeforeSourceStateMessageEmitted=23,maxSecondsBeforeSourceStateMessageEmitted=23,maxSecondsBetweenStateMessageEmittedandCommitted=4,meanSecondsBetweenStateMessageEmittedandCommitted=4,recordsEmitted=2336,recordsCommitted=2336,replicationEndTime=1683202275159,replicationStartTime=1683202234047,sourceReadEndTime=1683202270680,sourceReadStartTime=1683202234099,sourceStateMessagesEmitted=1,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@77b84790[streamName=ticket_comments,streamNamespace=zendesk,stats=io.airbyte.config.SyncStats@3798e743[bytesCommitted=6955374,bytesEmitted=6955374,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=<null>,estimatedRecords=<null>,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=2336,recordsCommitted=2336,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,additionalProperties={}],additionalProperties={}]],additionalProperties={}],normalizationSummary=<null>,webhookOperationSummary=<null>,state=<null>,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@42b78588[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@6b94eea6[stream=io.airbyte.protocol.models.AirbyteStream@5e25a083[name=ticket_comments,jsonSchema={"type":["null","object"],"properties":{"metadata":{"type":["null","object"],"properties":{"notifications_suppressed_for":{"type":["null","array"],"items":{"type":["null","integer"]}},"system":{"type":["null","object"],"properties":{"raw_email_identifier":{"type":["null","string"]},"latitude":{"type":["null","number"]},"client":{"type":["null","string"]},"json_email_identifier":{"type":["null","string"]},"location":{"type":["null","string"]},"message_id":{"type":["null","string"]},"ip_address":{"type":["null","string"]},"longitude":{"type":["null","number"]}}},"trusted":{"type":["null","boolean"]},"custom":{"additionalProperties":true,"type":["null","object"],"properties":{}},"flags":{"type":["null","array"],"items":{"type":["null","integer"]}},"flags_options":{"type":["null","object"],"properties":{"11":{"type":["null","object"],"properties":{"trusted":{"type":["null","boolean"]},"message":{"type":["null","object"],"properties":{"user":{"type":["null","string"]}}}}},"2":{"type":["null","object"],"properties":{"trusted":{"type":["null","boolean"]}}}}}}},"attachments":{"type":["null","array"],"items":{"type":["null","object"],"properties":{"size":{"type":["null","integer"]},"inline":{"type":["null","boolean"]},"content_type":{"type":["null","string"]},"mapped_content_url":{"type":["null","string"]},"file_name":{"type":["null","string"]},"width":{"type":["null","integer"]},"content_url":{"type":["null","string"]},"id":{"type":["null","integer"]},"thumbnails":{"type":["null","array"],"items":{"type":["null","object"],"properties":{"size":{"type":["null","integer"]},"inline":{"type":["null","boolean"]},"content_type":{"type":["null","string"]},"mapped_content_url":{"type":["null","string"]},"file_name":{"type":["null","string"]},"width":{"type":["null","integer"]},"content_url":{"type":["null","string"]},"id":{"type":["null","integer"]},"url":{"type":["null","string"]},"height":{"type":["null","integer"]}}}},"url":{"type":["null","string"]},"height":{"type":["null","integer"]}}}},"html_body":{"type":["null","string"]},"created_at":{"format":"date-time","type":["null","string"]},"plain_body":{"type":["null","string"]},"body":{"type":["null","string"]},"type":{"type":["null","string"]},"ticket_id":{"type":["null","integer"]},"via":{"type":["null","object"],"properties":{"channel":{"type":["null","string"]},"source":{"type":["null","object"],"properties":{"rel":{"type":["null","string"]},"from":{"type":["null","object"],"properties":{"address":{"type":["null","string"]},"deleted":{"type":["null","boolean"]},"ticket_ids":{"type":["null","array"],"items":{"type":["null","integer"]}},"subject":{"type":["null","string"]},"name":{"type":["null","string"]},"original_recipients":{"type":["null","array"],"items":{"type":["null","string"]}},"id":{"type":["null","integer"]},"title":{"type":["null","string"]},"ticket_id":{"type":["null","integer"]}}},"to":{"type":["null","object"],"properties":{"address":{"type":["null","string"]},"name":{"type":["null","string"]}}}}}}},"via_reference_id":{"type":["null","integer"]},"audit_id":{"type":["null","integer"]},"public":{"type":["null","boolean"]},"id":{"type":["null","integer"]},"author_id":{"type":["null","integer"]},"timestamp":{"type":["null","integer"]}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created_at],sourceDefinedPrimaryKey=[[id]],namespace=zendesk,additionalProperties={}],syncMode=incremental,cursorField=[created_at],destinationSyncMode=append_dedup,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}],failures=[],commitStateAsap=true,additionalProperties={}]
2023-05-04 12:11:15 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):162 - Sync summary length: 5634
2023-05-04 12:11:15 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):307 - Stopping temporal heartbeating...
2023-05-04 12:11:15 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):307 - Stopping temporal heartbeating...
2023-05-04 12:11:15 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=37022573-580e-3904-908b-6b4862596afa, activityType=Normalize, attempt=1
java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "String.split(String, int)" because the return value of "io.airbyte.persistence.job.models.IntegrationLauncherConfig.getNormalizationDockerImage()" is null
    at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:305) ~[io.airbyte-airbyte-commons-temporal-0.44.4.jar:?]
    at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:116) ~[io.airbyte-airbyte-workers-0.44.4.jar:?]
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "String.split(String, int)" because the return value of "io.airbyte.persistence.job.models.IntegrationLauncherConfig.getNormalizationDockerImage()" is null
    at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.getNormalizationImageTag(NormalizationActivityImpl.java:228) ~[io.airbyte-airbyte-workers-0.44.4.jar:?]
    at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalizationSupportsV1DataTypes(NormalizationActivityImpl.java:218) ~[io.airbyte-airbyte-workers-0.44.4.jar:?]
    at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:124) ~[io.airbyte-airbyte-workers-0.44.4.jar:?]
    at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:300) ~[io.airbyte-airbyte-commons-temporal-0.44.4.jar:?]
    ... 14 more

Logs

logs_failed_ticket_comments.log

emilmar commented 1 year ago

Solved it by reinstalling Docker :)

oleg-savko commented 1 year ago

Have the same problem, nothing worked :((

How to fix this exception with normalization?