Pipedrive Normalization Error Person Table: Invalid timestamp

[martinibach]

Enviroment

• Airbyte version: 0.29-21
• OS Version / Instance: Ubuntu 20.04
• Deployment: Docker 19.03.12
• Source Connector and version: Pipedrive 0.1.3
• Destination Connector and version: BigQuery 0.4.0
• Severity: High
• Step where error happened: Sync job

Current Behavior

Person table is not synched to BigQuery and normalisation fails with error because of invalid timestamp.

Expected Behavior

Normalisation should be processed and data should be synched to table in BigQuery.

Logs

LOG

``` 2021-09-23 19:06:42 INFO () WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-23 19:06:43 INFO () TemporalAttemptExecution(get):114 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-23 19:06:43 WARN () Databases(createPostgresDatabaseWithRetry):58 - Waiting for database to become available... 2021-09-23 19:06:43 INFO () JobsDatabaseInstance(lambda$static$2):45 - Testing if jobs database is ready... 2021-09-23 19:06:44 INFO () Databases(createPostgresDatabaseWithRetry):75 - Database available! 2021-09-23 19:06:44 WARN () JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-09-23 19:06:44 INFO () DefaultReplicationWorker(run):102 - start sync worker. job id: 18 attempt id: 0 2021-09-23 19:06:44 INFO () DefaultReplicationWorker(run):111 - configured sync modes: {null.persons=full_refresh - append} 2021-09-23 19:06:44 INFO () DefaultAirbyteDestination(start):78 - Running destination... 2021-09-23 19:06:44 INFO () LineGobbler(voidCall):85 - Checking if airbyte/destination-bigquery:0.4.0 exists... 2021-09-23 19:06:44 INFO () LineGobbler(voidCall):85 - airbyte/destination-bigquery:0.4.0 was found locally. 2021-09-23 19:06:44 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/18/0 --network host --log-driver none airbyte/destination-bigquery:0.4.0 write --config destination_config.json --catalog destination_catalog.json 2021-09-23 19:06:44 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-pipedrive:0.1.3 exists... 2021-09-23 19:06:44 INFO () LineGobbler(voidCall):85 - airbyte/source-pipedrive:0.1.3 was found locally. 2021-09-23 19:06:44 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/18/0 --network host --log-driver none airbyte/source-pipedrive:0.1.3 read --config source_config.json --catalog source_catalog.json 2021-09-23 19:06:44 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):246 - Destination output thread started. 2021-09-23 19:06:44 INFO () DefaultReplicationWorker(run):139 - Waiting for source thread to join. 2021-09-23 19:06:44 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):210 - Replication thread started. 2021-09-23 19:06:45 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Starting syncing SourcePipedrive 2021-09-23 19:06:45 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: persons 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.d.b.BigQueryDestination(main):356 - {} - starting destination: class io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.d.b.BigQueryDestination(getLoadingMethod):331 - {} - Selected loading method is set to: STANDARD 2021-09-23 19:06:47 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:47 INFO i.a.i.d.b.BigQueryDestination(isKeepFilesInGcs):344 - {} - All tmp files will be removed from GCS when migration is finished 2021-09-23 19:06:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:49 INFO i.a.i.d.b.BigQueryUtils(createTable):112 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_tmp_cpj_pipedrive_persons}} created successfully 2021-09-23 19:06:51 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 761 records from persons stream 2021-09-23 19:06:51 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Finished syncing SourcePipedrive 2021-09-23 19:06:51 INFO () DefaultReplicationWorker(run):141 - Source thread complete. 2021-09-23 19:06:51 INFO () DefaultReplicationWorker(run):142 - Waiting for destination thread to join. 2021-09-23 19:06:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:52 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded. 2021-09-23 19:06:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:52 INFO i.a.i.d.b.BigQueryRecordConsumer(close):163 - {} - Started closing all connections 2021-09-23 19:06:53 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:53 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):278 - {} - Waiting for jobs to be finished/closed 2021-09-23 19:06:55 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:55 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):295 - {} - Migration finished with no explicit errors. Copying data from tmp tables to permanent 2021-09-23 19:06:56 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:56 INFO i.a.i.d.b.BigQueryRecordConsumer(copyTable):359 - {} - successfully copied tmp table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_tmp_cpj_pipedrive_persons}} to final table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_raw_pipedrive_persons}} 2021-09-23 19:06:56 INFO () JsonSchemaValidator(test):76 - JSON schema validation failed. errors: $: null found, object expected 2021-09-23 19:06:56 ERROR () DefaultAirbyteStreamFactory(lambda$create$1):83 - Validation failed: null 2021-09-23 19:06:56 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:56 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):307 - {} - Removing tmp tables... 2021-09-23 19:06:56 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:56 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):309 - {} - Finishing destination process...completed 2021-09-23 19:06:56 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:56 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-23 19:06:56 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-23 19:06:56 INFO i.a.i.d.b.BigQueryDestination(main):358 - {} - completed destination: class io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-23 19:06:56 INFO () DefaultReplicationWorker(run):144 - Destination thread complete. 2021-09-23 19:06:56 INFO () DefaultReplicationWorker(run):172 - sync summary: io.airbyte.config.ReplicationAttemptSummary@297fad54[status=completed,recordsSynced=761,bytesSynced=1300241,startTime=1632424004303,endTime=1632424016786] 2021-09-23 19:06:56 INFO () DefaultReplicationWorker(run):181 - Source did not output any state messages 2021-09-23 19:06:56 WARN () DefaultReplicationWorker(run):192 - State capture: No state retained. 2021-09-23 19:06:56 INFO () TemporalAttemptExecution(get):135 - Stopping cancellation check scheduling... 2021-09-23 19:06:56 INFO () SyncWorkflow$ReplicationActivityImpl(replicate):184 - sync summary: io.airbyte.config.StandardSyncOutput@6242b963[standardSyncSummary=io.airbyte.config.StandardSyncSummary@1f6fd0eb[status=completed,recordsSynced=761,bytesSynced=1300241,startTime=1632424004303,endTime=1632424016786],state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@14045171[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@106420ce[stream=io.airbyte.protocol.models.AirbyteStream@679a0a00[name=pipedrive_persons,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":["null","integer"]},"name":{"type":["null","string"]},"email":{"type":["null","array"]},"label":{"type":["null","integer"]},"phone":{"type":["null","array"]},"org_id":{"type":["null","integer"]},"add_time":{"type":["null","string"],"format":"date-time"},"cc_email":{"type":["null","string"]},"org_name":{"type":["null","string"]},"owner_id":{"type":["null","integer"]},"last_name":{"type":["null","string"]},"company_id":{"type":["null","integer"]},"first_char":{"type":["null","string"]},"first_name":{"type":["null","string"]},"owner_name":{"type":["null","string"]},"picture_id":{"type":["null","object"],"properties":{"value":{"type":["null","integer"]},"item_id":{"type":["null","integer"]},"add_time":{"type":["null","string"],"format":"date-time"},"pictures":{"type":["null","object"],"properties":{"128":{"type":["null","string"]},"512":{"type":["null","string"]}}},"item_type":{"type":["null","string"]},"active_flag":{"type":["null","boolean"]},"update_time":{"type":["null","string"]},"added_by_user_id":{"type":["null","integer"]}}},"visible_to":{"type":["null","string"]},"active_flag":{"type":["null","boolean"]},"files_count":{"type":["null","integer"]},"notes_count":{"type":["null","integer"]},"update_time":{"type":["null","string"],"format":"date-time"},"followers_count":{"type":["null","integer"]},"won_deals_count":{"type":["null","integer"]},"activities_count":{"type":["null","integer"]},"last_activity_id":{"type":["null","integer"]},"lost_deals_count":{"type":["null","integer"]},"next_activity_id":{"type":["null","integer"]},"open_deals_count":{"type":["null","integer"]},"closed_deals_count":{"type":["null","integer"]},"last_activity_date":{"type":["null","string"],"format":"date-time"},"next_activity_date":{"type":["null","string"],"format":"date-time"},"next_activity_time":{"type":["null","string"],"format":"date-time"},"email_messages_count":{"type":["null","integer"]},"done_activities_count":{"type":["null","integer"]},"last_incoming_mail_time":{"type":["null","string"],"format":"date-time"},"last_outgoing_mail_time":{"type":["null","string"],"format":"date-time"},"related_won_deals_count":{"type":["null","integer"]},"undone_activities_count":{"type":["null","integer"]},"related_lost_deals_count":{"type":["null","integer"]},"related_open_deals_count":{"type":["null","integer"]},"related_closed_deals_count":{"type":["null","integer"]},"participant_open_deals_count":{"type":["null","integer"]},"participant_closed_deals_count":{"type":["null","integer"]}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[update_time],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[update_time],destinationSyncMode=append,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}]] 2021-09-23 19:06:56 INFO () TemporalAttemptExecution(get):114 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-23 19:06:56 WARN () Databases(createPostgresDatabaseWithRetry):58 - Waiting for database to become available... 2021-09-23 19:06:56 INFO () JobsDatabaseInstance(lambda$static$2):45 - Testing if jobs database is ready... 2021-09-23 19:06:56 INFO () Databases(createPostgresDatabaseWithRetry):75 - Database available! 2021-09-23 19:06:56 INFO () DefaultNormalizationWorker(run):63 - Running normalization. 2021-09-23 19:06:56 INFO () LineGobbler(voidCall):85 - Checking if airbyte/normalization:0.1.44 exists... 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - airbyte/normalization:0.1.44 was found locally. 2021-09-23 19:06:57 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/18/0/normalize --network host --log-driver none airbyte/normalization:0.1.44 run --integration-type bigquery --config destination_config.json --catalog destination_catalog.json 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Running: transform-config --config destination_config.json --integration-type bigquery --out /data/18/0/normalize 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Namespace(config='destination_config.json', integration_type=, out='/data/18/0/normalize') 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - transform_bigquery 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Running: transform-catalog --integration-type bigquery --profile-config-dir /data/18/0/normalize --catalog destination_catalog.json --out /data/18/0/normalize/models/generated/ --json-column _airbyte_data 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Processing destination_catalog.json... 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab1.sql from pipedrive_persons 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab2.sql from pipedrive_persons 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab3.sql from pipedrive_persons 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons.sql from pipedrive_persons 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab1.sql from pipedrive_persons/picture_id 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab2.sql from pipedrive_persons/picture_id 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab3.sql from pipedrive_persons/picture_id 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons_picture_id.sql from pipedrive_persons/picture_id 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab1.sql from pipedrive_persons/picture_id/pictures 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab2.sql from pipedrive_persons/picture_id/pictures 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab3.sql from pipedrive_persons/picture_id/pictures 2021-09-23 19:06:57 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons_picture_id_pictures.sql from pipedrive_persons/picture_id/pictures 2021-09-23 19:06:57 ERROR () LineGobbler(voidCall):85 - cat: /data/18/0/normalize/localsshport.json: No such file or directory 2021-09-23 19:06:58 INFO () LineGobbler(voidCall):85 - Running with dbt=0.19.0 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources. 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - There are 1 unused configuration paths: 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - - models.airbyte_utils.generated.airbyte_views 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - Found 12 models, 0 tests, 0 snapshots, 0 analyses, 423 macros, 0 operations, 0 seed files, 1 source, 0 exposures 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - 19:07:01 | Concurrency: 32 threads (target='prod') 2021-09-23 19:07:01 INFO () LineGobbler(voidCall):85 - 19:07:01 | 2021-09-23 19:07:02 INFO () LineGobbler(voidCall):85 - 19:07:02 | 1 of 3 START table model pipedrive.pipedrive_persons......................................................... [RUN] 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 19:07:06 | 1 of 3 ERROR creating table model pipedrive.pipedrive_persons................................................ [ERROR in 4.50s] 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 19:07:06 | 2 of 3 SKIP relation pipedrive.pipedrive_persons_picture_id.................................................. [SKIP] 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 19:07:06 | 3 of 3 SKIP relation pipedrive.pipedrive_persons_picture_id_pictures......................................... [SKIP] 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 19:07:06 | 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 19:07:06 | Finished running 3 table models in 5.45s. 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - Completed with 1 error and 0 warnings: 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - Database Error in model pipedrive_persons (models/generated/airbyte_tables/pipedrive/pipedrive_persons.sql) 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - Invalid timestamp: '08:30:00' 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/pipedrive/pipedrive_persons.sql 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - 2021-09-23 19:07:06 INFO () LineGobbler(voidCall):85 - Done. PASS=0 WARN=0 ERROR=1 SKIP=2 TOTAL=3 2021-09-23 19:07:07 INFO () TemporalAttemptExecution(lambda$getWorkerThread$1):168 - Completing future exceptionally... io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:77) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:74) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 3 more Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:151) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:62) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] 2021-09-23 19:07:07 INFO () TemporalAttemptExecution(get):135 - Stopping cancellation check scheduling... 2021-09-23 19:07:07 WARN () POJOActivityTaskHandler$POJOActivityImplementation(execute):243 - Activity failure. ActivityId=9f66c9e9-28a0-308b-b604-d17a9cef348d, activityType=Normalize, attempt=1 java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2063) ~[?:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:133) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.SyncWorkflow$NormalizationActivityImpl.normalize(SyncWorkflow.java:291) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?] at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:277) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:216) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:181) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:192) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:154) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.0.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:77) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 1 more Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:74) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 1 more Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:151) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:62) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] ```

Steps to Reproduce

1. Setup Pipedrive to BigQuery Connector
2. Export Person Table from Pipedrive with default normalization
3. Start Sync job

Are you willing to submit a PR?

No

[gaart]

@martin-restack

We just merged this bugfix into master and released a new version of the connector.

Upgrade your connector to version 0.1.5 and get started. To upgrade your connector version, go to the admin panel in the left hand side of the UI, find this connector in the list, and input the latest connector version.

Please let us know if you have any further questions.

Enjoy!

[martinibach]

@gaart thank you :-) Unfortunately it still doesn't seem to work. The logs also show, that I upgraded to 0.1.5 and still encounter the issue.

Logs

LOG

``` 2021-09-29 18:10:59 INFO () WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-29 18:10:59 INFO () TemporalAttemptExecution(get):114 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-29 18:10:59 WARN () Databases(createPostgresDatabaseWithRetry):58 - Waiting for database to become available... 2021-09-29 18:10:59 INFO () JobsDatabaseInstance(lambda$static$2):45 - Testing if jobs database is ready... 2021-09-29 18:10:59 INFO () Databases(createPostgresDatabaseWithRetry):75 - Database available! 2021-09-29 18:10:59 INFO () DefaultReplicationWorker(run):102 - start sync worker. job id: 21 attempt id: 1 2021-09-29 18:10:59 INFO () DefaultReplicationWorker(run):111 - configured sync modes: {null.persons=full_refresh - append} 2021-09-29 18:10:59 INFO () DefaultAirbyteDestination(start):78 - Running destination... 2021-09-29 18:10:59 INFO () LineGobbler(voidCall):85 - Checking if airbyte/destination-bigquery:0.4.0 exists... 2021-09-29 18:10:59 INFO () LineGobbler(voidCall):85 - airbyte/destination-bigquery:0.4.0 was found locally. 2021-09-29 18:10:59 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/21/1 --network host --log-driver none airbyte/destination-bigquery:0.4.0 write --config destination_config.json --catalog destination_catalog.json 2021-09-29 18:10:59 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-pipedrive:0.1.5 exists... 2021-09-29 18:10:59 INFO () LineGobbler(voidCall):85 - airbyte/source-pipedrive:0.1.5 was found locally. 2021-09-29 18:10:59 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/21/1 --network host --log-driver none airbyte/source-pipedrive:0.1.5 read --config source_config.json --catalog source_catalog.json --state input_state.json 2021-09-29 18:10:59 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):246 - Destination output thread started. 2021-09-29 18:10:59 INFO () DefaultReplicationWorker(run):139 - Waiting for source thread to join. 2021-09-29 18:10:59 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):210 - Replication thread started. 2021-09-29 18:11:01 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Starting syncing SourcePipedrive 2021-09-29 18:11:01 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: persons 2021-09-29 18:11:01 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:01 INFO i.a.i.d.b.BigQueryDestination(main):356 - {} - starting destination: class io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.d.b.BigQueryDestination(getLoadingMethod):331 - {} - Selected loading method is set to: STANDARD 2021-09-29 18:11:02 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:02 INFO i.a.i.d.b.BigQueryDestination(isKeepFilesInGcs):344 - {} - All tmp files will be removed from GCS when migration is finished 2021-09-29 18:11:04 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:04 INFO i.a.i.d.b.BigQueryUtils(createTable):112 - {} - Table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_tmp_gmw_pipedrive_persons}} created successfully 2021-09-29 18:11:05 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Read 772 records from persons stream 2021-09-29 18:11:05 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Finished syncing SourcePipedrive 2021-09-29 18:11:05 INFO () DefaultReplicationWorker(run):141 - Source thread complete. 2021-09-29 18:11:05 INFO () DefaultReplicationWorker(run):142 - Waiting for destination thread to join. 2021-09-29 18:11:05 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:05 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded. 2021-09-29 18:11:05 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:05 INFO i.a.i.d.b.BigQueryRecordConsumer(close):163 - {} - Started closing all connections 2021-09-29 18:11:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:06 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):278 - {} - Waiting for jobs to be finished/closed 2021-09-29 18:11:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:50 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):295 - {} - Migration finished with no explicit errors. Copying data from tmp tables to permanent 2021-09-29 18:11:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:54 INFO i.a.i.d.b.BigQueryRecordConsumer(copyTable):359 - {} - successfully copied tmp table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_tmp_gmw_pipedrive_persons}} to final table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=pipedrive, tableId=_airbyte_raw_pipedrive_persons}} 2021-09-29 18:11:54 INFO () JsonSchemaValidator(test):76 - JSON schema validation failed. errors: $: null found, object expected 2021-09-29 18:11:54 ERROR () DefaultAirbyteStreamFactory(lambda$create$1):83 - Validation failed: null 2021-09-29 18:11:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:54 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):307 - {} - Removing tmp tables... 2021-09-29 18:11:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:54 INFO i.a.i.d.b.BigQueryRecordConsumer(closeNormalBigqueryStreams):309 - {} - Finishing destination process...completed 2021-09-29 18:11:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:54 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-29 18:11:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-09-29 18:11:54 INFO i.a.i.d.b.BigQueryDestination(main):358 - {} - completed destination: class io.airbyte.integrations.destination.bigquery.BigQueryDestination 2021-09-29 18:11:54 INFO () DefaultReplicationWorker(run):144 - Destination thread complete. 2021-09-29 18:11:54 INFO () DefaultReplicationWorker(run):172 - sync summary: io.airbyte.config.ReplicationAttemptSummary@5d24898[status=completed,recordsSynced=772,bytesSynced=1319290,startTime=1632939059840,endTime=1632939114423] 2021-09-29 18:11:54 INFO () DefaultReplicationWorker(run):181 - Source did not output any state messages 2021-09-29 18:11:54 WARN () DefaultReplicationWorker(run):189 - State capture: No new state, falling back on input state: io.airbyte.config.State@9093398[state={}] 2021-09-29 18:11:54 INFO () TemporalAttemptExecution(get):135 - Stopping cancellation check scheduling... 2021-09-29 18:11:54 INFO () SyncWorkflow$ReplicationActivityImpl(replicate):184 - sync summary: io.airbyte.config.StandardSyncOutput@1dce75b6[standardSyncSummary=io.airbyte.config.StandardSyncSummary@2865e1c2[status=completed,recordsSynced=772,bytesSynced=1319290,startTime=1632939059840,endTime=1632939114423],state=io.airbyte.config.State@9093398[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@57402fc3[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@770d345b[stream=io.airbyte.protocol.models.AirbyteStream@19808cc9[name=pipedrive_persons,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":["null","integer"]},"name":{"type":["null","string"]},"email":{"type":["null","array"]},"label":{"type":["null","integer"]},"phone":{"type":["null","array"]},"org_id":{"type":["null","integer"]},"add_time":{"type":["null","string"],"format":"date-time"},"cc_email":{"type":["null","string"]},"org_name":{"type":["null","string"]},"owner_id":{"type":["null","integer"]},"last_name":{"type":["null","string"]},"company_id":{"type":["null","integer"]},"first_char":{"type":["null","string"]},"first_name":{"type":["null","string"]},"owner_name":{"type":["null","string"]},"picture_id":{"type":["null","object"],"properties":{"value":{"type":["null","integer"]},"item_id":{"type":["null","integer"]},"add_time":{"type":["null","string"],"format":"date-time"},"pictures":{"type":["null","object"],"properties":{"128":{"type":["null","string"]},"512":{"type":["null","string"]}}},"item_type":{"type":["null","string"]},"active_flag":{"type":["null","boolean"]},"update_time":{"type":["null","string"]},"added_by_user_id":{"type":["null","integer"]}}},"visible_to":{"type":["null","string"]},"active_flag":{"type":["null","boolean"]},"files_count":{"type":["null","integer"]},"notes_count":{"type":["null","integer"]},"update_time":{"type":["null","string"],"format":"date-time"},"followers_count":{"type":["null","integer"]},"won_deals_count":{"type":["null","integer"]},"activities_count":{"type":["null","integer"]},"last_activity_id":{"type":["null","integer"]},"lost_deals_count":{"type":["null","integer"]},"next_activity_id":{"type":["null","integer"]},"open_deals_count":{"type":["null","integer"]},"closed_deals_count":{"type":["null","integer"]},"last_activity_date":{"type":["null","string"],"format":"date-time"},"next_activity_date":{"type":["null","string"],"format":"date-time"},"next_activity_time":{"type":["null","string"],"format":"date-time"},"email_messages_count":{"type":["null","integer"]},"done_activities_count":{"type":["null","integer"]},"last_incoming_mail_time":{"type":["null","string"],"format":"date-time"},"last_outgoing_mail_time":{"type":["null","string"],"format":"date-time"},"related_won_deals_count":{"type":["null","integer"]},"undone_activities_count":{"type":["null","integer"]},"related_lost_deals_count":{"type":["null","integer"]},"related_open_deals_count":{"type":["null","integer"]},"related_closed_deals_count":{"type":["null","integer"]},"participant_open_deals_count":{"type":["null","integer"]},"participant_closed_deals_count":{"type":["null","integer"]}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[update_time],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[update_time],destinationSyncMode=append,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}]] 2021-09-29 18:11:54 INFO () TemporalAttemptExecution(get):114 - Executing worker wrapper. Airbyte version: 0.29.21-alpha 2021-09-29 18:11:54 WARN () Databases(createPostgresDatabaseWithRetry):58 - Waiting for database to become available... 2021-09-29 18:11:54 INFO () JobsDatabaseInstance(lambda$static$2):45 - Testing if jobs database is ready... 2021-09-29 18:11:54 INFO () Databases(createPostgresDatabaseWithRetry):75 - Database available! 2021-09-29 18:11:54 INFO () DefaultNormalizationWorker(run):63 - Running normalization. 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - Checking if airbyte/normalization:0.1.44 exists... 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - airbyte/normalization:0.1.44 was found locally. 2021-09-29 18:11:54 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/21/1/normalize --network host --log-driver none airbyte/normalization:0.1.44 run --integration-type bigquery --config destination_config.json --catalog destination_catalog.json 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - Running: transform-config --config destination_config.json --integration-type bigquery --out /data/21/1/normalize 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - Namespace(config='destination_config.json', integration_type=, out='/data/21/1/normalize') 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - transform_bigquery 2021-09-29 18:11:54 INFO () LineGobbler(voidCall):85 - Running: transform-catalog --integration-type bigquery --profile-config-dir /data/21/1/normalize --catalog destination_catalog.json --out /data/21/1/normalize/models/generated/ --json-column _airbyte_data 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Processing destination_catalog.json... 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab1.sql from pipedrive_persons 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab2.sql from pipedrive_persons 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_ab3.sql from pipedrive_persons 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons.sql from pipedrive_persons 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab1.sql from pipedrive_persons/picture_id 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab2.sql from pipedrive_persons/picture_id 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_ab3.sql from pipedrive_persons/picture_id 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons_picture_id.sql from pipedrive_persons/picture_id 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab1.sql from pipedrive_persons/picture_id/pictures 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab2.sql from pipedrive_persons/picture_id/pictures 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_ctes/pipedrive/pipedrive_persons_picture_id_pictures_ab3.sql from pipedrive_persons/picture_id/pictures 2021-09-29 18:11:55 INFO () LineGobbler(voidCall):85 - Generating airbyte_tables/pipedrive/pipedrive_persons_picture_id_pictures.sql from pipedrive_persons/picture_id/pictures 2021-09-29 18:11:55 ERROR () LineGobbler(voidCall):85 - cat: /data/21/1/normalize/localsshport.json: No such file or directory 2021-09-29 18:11:56 INFO () LineGobbler(voidCall):85 - Running with dbt=0.19.0 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources. 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - There are 1 unused configuration paths: 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - - models.airbyte_utils.generated.airbyte_views 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - Found 12 models, 0 tests, 0 snapshots, 0 analyses, 423 macros, 0 operations, 0 seed files, 1 source, 0 exposures 2021-09-29 18:11:58 INFO () LineGobbler(voidCall):85 - 2021-09-29 18:11:59 INFO () LineGobbler(voidCall):85 - 18:11:59 | Concurrency: 32 threads (target='prod') 2021-09-29 18:11:59 INFO () LineGobbler(voidCall):85 - 18:11:59 | 2021-09-29 18:11:59 INFO () LineGobbler(voidCall):85 - 18:11:59 | 1 of 3 START table model pipedrive.pipedrive_persons......................................................... [RUN] 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 18:12:04 | 1 of 3 ERROR creating table model pipedrive.pipedrive_persons................................................ [ERROR in 5.23s] 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 18:12:04 | 2 of 3 SKIP relation pipedrive.pipedrive_persons_picture_id.................................................. [SKIP] 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 18:12:04 | 3 of 3 SKIP relation pipedrive.pipedrive_persons_picture_id_pictures......................................... [SKIP] 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 18:12:04 | 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 18:12:04 | Finished running 3 table models in 5.99s. 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - Completed with 1 error and 0 warnings: 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - Database Error in model pipedrive_persons (models/generated/airbyte_tables/pipedrive/pipedrive_persons.sql) 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - Invalid timestamp: '08:30:00' 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/pipedrive/pipedrive_persons.sql 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - 2021-09-29 18:12:04 INFO () LineGobbler(voidCall):85 - Done. PASS=0 WARN=0 ERROR=1 SKIP=2 TOTAL=3 2021-09-29 18:12:05 INFO () TemporalAttemptExecution(lambda$getWorkerThread$1):168 - Completing future exceptionally... io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:77) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:74) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 3 more Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:151) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:62) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] 2021-09-29 18:12:05 INFO () TemporalAttemptExecution(get):135 - Stopping cancellation check scheduling... 2021-09-29 18:12:05 WARN () POJOActivityTaskHandler$POJOActivityImplementation(execute):243 - Activity failure. ActivityId=5bd6e5ec-80c9-3125-a92e-b799c10a33bc, activityType=Normalize, attempt=1 java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Normalization Failed. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2063) ~[?:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:133) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.SyncWorkflow$NormalizationActivityImpl.normalize(SyncWorkflow.java:291) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?] at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:277) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:216) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:181) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:192) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:154) ~[temporal-sdk-1.0.4.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.0.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:77) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 1 more Caused by: io.airbyte.workers.WorkerException: Normalization Failed. at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:74) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] ... 1 more Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:151) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:62) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:165) ~[io.airbyte-airbyte-workers-0.29.21-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] ```

[martinibach]

@gaart did you have time to look at this again? :)

[marcosmarxm]

@gaart opening the issue again because is not solved yet. I run our integration account and normalization works, but probably the columns @martin-restack has problem, our integration account doesn't have any data.

[gaart]

Hi @martin-restack Could you try to run it again with the latest 0.1.6 version pulled from the docker hub registry? In the logs the schema looks different from what we expect to have there.

[martinibach]

@gaart I deployed it now in version 0.30.20-alpha on Kubernetes on Digital Ocean and it's working with 0.1.6 version properly. logs-35-0 (1).txt