search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.89k stars 3.82k forks source link

[source-hubspot] missing data for contacts stream #40591

Open ASmith12345 opened 2 weeks ago

ASmith12345 commented 2 weeks ago

Connector Name

source-hubspot destination-databricks-lakehouse

Connector Version

4.2.7

What step the error happened?

During the job run

Relevant information

we have had missing hubspot contacts 3 times in the last 2 weeks. The contacts typically have been created at around 8am the day before and our airbyte job runs at around 4am.

this issue seems to be related to the same problem. https://github.com/airbytehq/airbyte/issues/34446

Relevant log output

2024-06-26 03:09:07 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):103 Write config: WriteConfig{streamName=contacts, namespace=hubspot_airbyte, outputSchemaName=hubspot_airbyte, tmpTableName=_airbyte_tmp_vhj_contacts, outputTableName=_airbyte_raw_contacts, syncMode=append}

2024-06-26 03:09:08 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream contacts. schema: hubspot_airbyte, table name: _airbyte_raw_contacts

2024-06-26 03:09:14 source > Marking stream contacts as STARTED
2024-06-26 03:09:14 source > Setting state of SourceHubspot stream to {'updatedAt': '2024-06-25T16:00:32.840747Z'}
2024-06-26 03:09:14 source > Syncing stream: contacts 
2024-06-26 03:09:14 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream subscription_license. schema: hubspot_airbyte, table name: _airbyte_raw_subscription_license
2024-06-26 03:09:14 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream cmqls. schema: hubspot_airbyte, table name: _airbyte_raw_cmqls
2024-06-26 03:09:14 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream agreements. schema: hubspot_airbyte, table name: _airbyte_raw_agreements
2024-06-26 03:09:14 source > Reading contacts associations of contact
2024-06-26 03:09:14 source > Reading companies associations of contact
2024-06-26 03:09:14 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream mql. schema: hubspot_airbyte, table name: _airbyte_raw_mql
2024-06-26 03:09:14 source > Marking stream contacts as RUNNING
2024-06-26 03:09:16 source > Reading contacts associations of contact
2024-06-26 03:09:16 source > Reading companies associations of contact
2024-06-26 03:09:18 source > Reading contacts associations of contact
2024-06-26 03:09:18 source > Reading companies associations of contact
2024-06-26 03:09:20 source > Reading contacts associations of contact
2024-06-26 03:09:20 source > Reading companies associations of contact
2024-06-26 03:09:22 source > Reading contacts associations of contact
2024-06-26 03:09:22 source > Reading companies associations of contact
2024-06-26 03:09:23 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:09:23 source > Reading contacts associations of contact
2024-06-26 03:09:24 source > Reading companies associations of contact
2024-06-26 03:09:25 source > Reading contacts associations of contact
2024-06-26 03:09:25 source > Reading companies associations of contact
2024-06-26 03:09:27 source > Reading contacts associations of contact
2024-06-26 03:09:27 source > Reading companies associations of contact
2024-06-26 03:09:29 source > Reading contacts associations of contact
2024-06-26 03:09:29 source > Reading companies associations of contact
2024-06-26 03:09:30 source > Reading contacts associations of contact
2024-06-26 03:09:30 source > Reading companies associations of contact
2024-06-26 03:09:32 source > Reading contacts associations of contact
2024-06-26 03:09:32 source > Reading companies associations of contact
2024-06-26 03:09:34 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):160 Preparing raw tables in destination completed.
2024-06-26 03:09:34 source > Reading contacts associations of contact
2024-06-26 03:09:34 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing companies: 188 records (24 MB)
2024-06-26 03:09:34 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 188
2024-06-26 03:09:35 source > Reading companies associations of contact
2024-06-26 03:09:36 source > Reading contacts associations of contact
2024-06-26 03:09:36 source > Reading companies associations of contact
2024-06-26 03:09:38 source > Reading contacts associations of contact
2024-06-26 03:09:38 source > Reading companies associations of contact
2024-06-26 03:09:40 source > Reading contacts associations of contact
2024-06-26 03:09:40 source > Reading companies associations of contact
2024-06-26 03:09:42 source > Reading contacts associations of contact
2024-06-26 03:09:42 source > Reading companies associations of contact
2024-06-26 03:09:43 source > Reading contacts associations of contact
2024-06-26 03:09:44 source > Reading companies associations of contact
2024-06-26 03:09:45 source > Reading contacts associations of contact
2024-06-26 03:09:45 source > Reading companies associations of contact
2024-06-26 03:09:47 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for companies
2024-06-26 03:09:47 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:09:47 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 19 records (24 MB)
2024-06-26 03:09:47 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 19
2024-06-26 03:09:47 source > Reading contacts associations of contact
2024-06-26 03:09:47 source > Reading companies associations of contact
2024-06-26 03:09:49 source > Reading contacts associations of contact
2024-06-26 03:09:49 source > Reading companies associations of contact
2024-06-26 03:09:51 source > Reading contacts associations of contact
2024-06-26 03:09:51 source > Reading companies associations of contact
2024-06-26 03:09:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:09:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing companies: 154 records (24 MB)
2024-06-26 03:09:52 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 154
2024-06-26 03:10:00 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for companies
2024-06-26 03:10:00 replication-orchestrator > starting state flush thread for connectionId 656be75f-94bc-47e7-bdd6-6140506be3c2
2024-06-26 03:10:00 source > Reading contacts associations of contact
2024-06-26 03:10:00 source > Reading companies associations of contact
2024-06-26 03:10:01 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:10:01 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:10:09 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:09 source > Reading contacts associations of contact
2024-06-26 03:10:09 source > Reading companies associations of contact
2024-06-26 03:10:09 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:10:09 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:10:17 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:17 source > Reading contacts associations of contact
2024-06-26 03:10:17 source > Reading companies associations of contact
2024-06-26 03:10:17 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:10:17 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:10:25 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:25 source > Reading contacts associations of contact
2024-06-26 03:10:25 source > Reading companies associations of contact
2024-06-26 03:10:25 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:10:25 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:10:33 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:33 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:10:33 source > Reading contacts associations of contact
2024-06-26 03:10:33 source > Reading companies associations of contact
2024-06-26 03:10:33 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 105 records (24 MB)
2024-06-26 03:10:33 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 105
2024-06-26 03:10:40 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:40 source > Reading contacts associations of contact
2024-06-26 03:10:40 source > Reading companies associations of contact
2024-06-26 03:10:40 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:10:40 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:10:48 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:48 source > Reading contacts associations of contact
2024-06-26 03:10:48 source > Reading companies associations of contact
2024-06-26 03:10:48 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:10:48 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:10:48 source > Reading contacts associations of contact
2024-06-26 03:10:48 source > Reading companies associations of contact
2024-06-26 03:10:56 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:10:56 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:10:56 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:10:56 source > Reading contacts associations of contact
2024-06-26 03:10:56 source > Reading companies associations of contact
2024-06-26 03:11:03 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:04 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:11:04 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:11:04 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:11:04 source > Reading contacts associations of contact
2024-06-26 03:11:04 source > Reading companies associations of contact
2024-06-26 03:11:11 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:12 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:11:12 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:11:12 source > Reading contacts associations of contact
2024-06-26 03:11:12 source > Reading companies associations of contact
2024-06-26 03:11:19 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:19 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:11:19 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:11:19 source > Reading contacts associations of contact
2024-06-26 03:11:19 source > Reading companies associations of contact
2024-06-26 03:11:26 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:27 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:11:27 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:11:27 source > Reading contacts associations of contact
2024-06-26 03:11:27 source > Reading companies associations of contact
2024-06-26 03:11:34 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:34 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:11:34 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:11:34 source > Reading contacts associations of contact
2024-06-26 03:11:34 source > Reading companies associations of contact
2024-06-26 03:11:41 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:41 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:11:42 source > Reading contacts associations of contact
2024-06-26 03:11:42 source > Reading companies associations of contact
2024-06-26 03:11:42 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:11:42 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:11:48 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:49 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:11:49 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:11:49 source > Reading contacts associations of contact
2024-06-26 03:11:49 source > Reading companies associations of contact
2024-06-26 03:11:56 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:11:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:11:57 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:11:57 source > Reading contacts associations of contact
2024-06-26 03:11:57 source > Reading companies associations of contact
2024-06-26 03:12:04 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:04 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:12:04 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:12:04 source > Reading contacts associations of contact
2024-06-26 03:12:04 source > Reading companies associations of contact
2024-06-26 03:12:12 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:12 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 105 records (24 MB)
2024-06-26 03:12:12 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 105
2024-06-26 03:12:12 source > Reading contacts associations of contact
2024-06-26 03:12:12 source > Reading companies associations of contact
2024-06-26 03:12:19 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:19 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:12:19 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:12:19 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:12:19 source > Reading contacts associations of contact
2024-06-26 03:12:19 source > Reading companies associations of contact
2024-06-26 03:12:26 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:27 source > Reading contacts associations of contact
2024-06-26 03:12:27 source > Reading companies associations of contact
2024-06-26 03:12:27 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:12:27 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:12:34 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:34 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:12:34 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:12:34 source > Reading contacts associations of contact
2024-06-26 03:12:34 source > Reading companies associations of contact
2024-06-26 03:12:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:44 source > Reading contacts associations of contact
2024-06-26 03:12:44 source > Reading companies associations of contact
2024-06-26 03:12:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:12:44 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:12:53 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:12:53 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:12:53 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:12:53 source > Reading contacts associations of contact
2024-06-26 03:12:53 source > Reading companies associations of contact
2024-06-26 03:13:02 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:02 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:13:02 source > Reading contacts associations of contact
2024-06-26 03:13:02 source > Reading companies associations of contact
2024-06-26 03:13:02 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 105 records (24 MB)
2024-06-26 03:13:02 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 105
2024-06-26 03:13:09 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:10 source > Reading contacts associations of contact
2024-06-26 03:13:10 source > Reading companies associations of contact
2024-06-26 03:13:10 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:13:10 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:13:17 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:17 source > Reading contacts associations of contact
2024-06-26 03:13:17 source > Reading companies associations of contact
2024-06-26 03:13:17 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:13:17 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:13:24 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:24 source > Reading contacts associations of contact
2024-06-26 03:13:24 source > Reading companies associations of contact
2024-06-26 03:13:24 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:13:24 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:13:31 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:31 source > Reading contacts associations of contact
2024-06-26 03:13:31 source > Reading companies associations of contact
2024-06-26 03:13:31 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:13:31 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:13:38 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:38 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:13:38 source > Reading contacts associations of contact
2024-06-26 03:13:38 source > Reading companies associations of contact
2024-06-26 03:13:38 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:13:38 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:13:45 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:46 source > Reading contacts associations of contact
2024-06-26 03:13:46 source > Reading companies associations of contact
2024-06-26 03:13:46 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:13:46 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:13:53 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:13:53 source > Reading contacts associations of contact
2024-06-26 03:13:53 source > Reading companies associations of contact
2024-06-26 03:13:53 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:13:53 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:14:00 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:00 source > Reading contacts associations of contact
2024-06-26 03:14:00 source > Reading companies associations of contact
2024-06-26 03:14:00 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:00 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:08 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:08 source > Reading contacts associations of contact
2024-06-26 03:14:08 source > Reading companies associations of contact
2024-06-26 03:14:08 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:08 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:16 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:16 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:14:16 source > Reading contacts associations of contact
2024-06-26 03:14:16 source > Reading companies associations of contact
2024-06-26 03:14:16 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:14:16 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:14:23 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:23 source > Reading contacts associations of contact
2024-06-26 03:14:23 source > Reading companies associations of contact
2024-06-26 03:14:23 replication-orchestrator > Records read: 5000 (289 MB)
2024-06-26 03:14:23 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:23 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:23 source > Reading contacts associations of contact
2024-06-26 03:14:23 source > Reading companies associations of contact
2024-06-26 03:14:30 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:30 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:30 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:30 source > Reading contacts associations of contact
2024-06-26 03:14:30 source > Reading companies associations of contact
2024-06-26 03:14:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:38 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:14:38 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:14:38 source > Reading contacts associations of contact
2024-06-26 03:14:38 source > Reading companies associations of contact
2024-06-26 03:14:45 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:45 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:14:45 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:45 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:45 source > Reading contacts associations of contact
2024-06-26 03:14:45 source > Reading companies associations of contact
2024-06-26 03:14:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:14:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:14:52 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:14:52 source > Reading contacts associations of contact
2024-06-26 03:14:52 source > Reading companies associations of contact
2024-06-26 03:14:59 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:00 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:15:00 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:15:00 source > Reading contacts associations of contact
2024-06-26 03:15:00 source > Reading companies associations of contact
2024-06-26 03:15:06 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:07 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:15:07 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:15:07 source > Reading contacts associations of contact
2024-06-26 03:15:07 source > Reading companies associations of contact
2024-06-26 03:15:14 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:14 source > Reading contacts associations of contact
2024-06-26 03:15:14 source > Reading companies associations of contact
2024-06-26 03:15:14 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:15:14 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:15:21 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:21 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 105 records (24 MB)
2024-06-26 03:15:21 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 105
2024-06-26 03:15:21 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:15:22 source > Reading contacts associations of contact
2024-06-26 03:15:22 source > Reading companies associations of contact
2024-06-26 03:15:29 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:29 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:15:29 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:15:30 source > Reading contacts associations of contact
2024-06-26 03:15:30 source > Reading companies associations of contact
2024-06-26 03:15:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:15:37 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:15:37 source > Reading contacts associations of contact
2024-06-26 03:15:37 source > Reading companies associations of contact
2024-06-26 03:15:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:44 source > Reading contacts associations of contact
2024-06-26 03:15:44 source > Reading companies associations of contact
2024-06-26 03:15:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:15:44 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:15:51 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:15:52 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:15:52 source > Reading contacts associations of contact
2024-06-26 03:15:52 source > Reading companies associations of contact
2024-06-26 03:15:59 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:15:59 source > Advancing bookmark for contacts stream from 2024-06-25T16:00:32.840747+00:00 to 2024-06-26T03:15:22.007000+00:00
2024-06-26 03:15:59 source > Read 6971 records from contacts stream
2024-06-26 03:15:59 source > Marking stream contacts as STOPPED
2024-06-26 03:15:59 source > Finished syncing contacts
2024-06-26 03:15:59 source > SourceHubspot runtimes:
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617

2024-06-26 03:15:59 replication-orchestrator > Source state message checksum is valid for stream _contacts.

2024-06-26 03:15:59 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:15:59 source > Marking stream deals as RUNNING
2024-06-26 03:16:06 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts

2024-06-26 03:16:06 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:16:06 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:16:13 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:16:13 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 105 records (24 MB)
2024-06-26 03:16:13 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 105
2024-06-26 03:16:21 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:16:21 source > Couldn't parse date/datetime string in nrr_date, trying to parse timestamp... Field value: 1740787200000. Ex: Unable to parse string [1740787200000]
2024-06-26 03:16:21 source > Couldn't parse date/datetime string in nrr_date, trying to parse timestamp... Field value: 1674518400000. Ex: Unable to parse string [1674518400000]
2024-06-26 03:16:21 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:16:21 source > Couldn't parse date/datetime string in nrr_date, trying to parse timestamp... Field value: 1663632000000. Ex: Unable to parse string [1663632000000]
2024-06-26 03:16:21 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103

2024-06-26 03:16:28 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts

2024-06-26 03:16:28 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:16:28 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:16:35 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:16:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:16:36 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:16:43 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts

2024-06-26 03:16:43 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:16:43 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:16:50 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:16:50 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:16:50 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:16:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:16:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:16:57 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
2024-06-26 03:17:04 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:05 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:17:05 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:17:05 replication-orchestrator > Source state message checksum is valid for stream _contacts.
2024-06-26 03:17:13 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:13 replication-orchestrator > Source state message checksum is valid for stream _deal_pipelines.
2024-06-26 03:17:14 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:17:14 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:17:22 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:22 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:17:22 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103
]
2024-06-26 03:17:30 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:30 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:17:30 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:17:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:17:37 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:17:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:44 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:17:44 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102
2024-06-26 03:17:51 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:52 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 101 records (24 MB)
2024-06-26 03:17:52 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 101

2024-06-26 03:17:59 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:17:59 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:17:59 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102

2024-06-26 03:18:06 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:18:06 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 104 records (24 MB)
2024-06-26 03:18:06 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 104
2024-06-26 03:18:13 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:18:13 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 103 records (24 MB)
2024-06-26 03:18:13 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 103

2024-06-26 03:18:20 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts
2024-06-26 03:18:21 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 102 records (24 MB)
2024-06-26 03:18:21 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 102

2024-06-26 03:18:21 source > Couldn't parse date/datetime string in nrr_date, trying to parse timestamp... Field value: 1498694400000. Ex: Unable to parse string [1498694400000]
2024-06-26 03:18:27 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts

2024-06-26 03:18:28 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing contacts: 25 records (24 MB)
2024-06-26 03:18:28 destination > INFO i.a.i.d.d.DatabricksSqlOperations(insertRecordsInternal):58 actual size of batch: 25
2024-06-26 03:18:31 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for contacts

Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
2024-06-26 03:33:51 source > Reading contacts associations of calls

Syncing stream contacts 0:06:39.402617
2024-06-26 03:33:51 source > {"status":"error","message":"This app hasn't been granted all required scopes to make this call. Read more about required scopes here: https://developers.hubspot.com/scopes.","correlationId":"aacec894-73cf-4d5a-83e2-f84ffc8aa7df","errors":[{"message":"One or more of the following scopes are required.","context":{"requiredScopes":["sales-email-read"]}}],"links":{"scopes":"https://developers.hubspot.com/scopes"},"category":"MISSING_SCOPES"}
2024-06-26 03:33:51 source > Skipped syncing stream 'engagements_emails' because it was unavailable. Unable to read engagements_emails stream. The endpoint https://api.hubapi.com/crm/v3/objects/emails?archived=false&associations=contacts&associations=deal&associations=company&associations=tickets&limit=100&properties=hs_all_assigned_business_unit_ids%2Chs_at_mentioned_owner_ids%2Chs_attachment_ids%2Chs_body_preview%2Chs_body_preview_html%2Chs_body_preview_is_truncated%2Chs_created_by%2Chs_created_by_user_id%2Chs_createdate%2Chs_direction_and_unique_id%2Chs_email_attached_video_id%2Chs_email_attached_video_name%2Chs_email_attached_video_opened%2Chs_email_attached_video_watched%2Chs_email_bcc_email%2Chs_email_bcc_firstname%2Chs_email_bcc_lastname%2Chs_email_bcc_raw%2Chs_email_bounce_error_detail_message%2Chs_email_bounce_error_detail_status_code%2Chs_email_cc_email%2Chs_email_cc_firstname%2Chs_email_cc_lastname%2Chs_email_cc_raw%2Chs_email_click_count%2Chs_email_direction%2Chs_email_encoded_email_associations_request%2Chs_email_error_message%2Chs_email_facsimile_send_id%2Chs_email_from_email%2Chs_email_from_firstname%2Chs_email_from_lastname%2Chs_email_from_raw%2Chs_email_has_inline_images_stripped%2Chs_email_headers%2Chs_email_html%2Chs_email_logged_from%2Chs_email_media_processing_status%2Chs_email_member_of_forwarded_subthread%2Chs_email_message_id%2Chs_email_migrated_via_portal_data_migration%2Chs_email_ms_teams_payload%2Chs_email_open_count%2Chs_email_pending_inline_image_ids%2Chs_email_post_send_status%2Chs_email_recipient_drop_reasons%2Chs_email_reply_count%2Chs_email_send_event_id%2Chs_email_send_event_id_created%2Chs_email_sender_email%2Chs_email_sender_firstname%2Chs_email_sender_lastname%2Chs_email_sender_raw%2Chs_email_sent_count%2Chs_email_sent_via%2Chs_email_status%2Chs_email_stripped_attachment_count%2Chs_email_subject%2Chs_email_text%2Chs_email_thread_id%2Chs_email_thread_summary%2Chs_email_to_email%2Chs_email_to_firstname%2Chs_email_to_lastname%2Chs_email_to_raw%2Chs_email_tracker_key%2Chs_email_validation_skipped%2Chs_engagement_source%2Chs_engagement_source_id%2Chs_follow_up_action%2Chs_gdpr_deleted%2Chs_lastmodifieddate%2Chs_merged_object_ids%2Chs_modified_by%2Chs_object_id%2Chs_object_source%2Chs_object_source_detail_1%2Chs_object_source_detail_2%2Chs_object_source_detail_3%2Chs_object_source_id%2Chs_object_source_label%2Chs_object_source_user_id%2Chs_owner_ids_bcc%2Chs_owner_ids_cc%2Chs_owner_ids_from%2Chs_owner_ids_to%2Chs_product_name%2Chs_queue_membership_ids%2Chs_read_only%2Chs_scs_association_status%2Chs_scs_audit_id%2Chs_shared_team_ids%2Chs_shared_user_ids%2Chs_timestamp%2Chs_unique_creation_key%2Chs_unique_id%2Chs_updated_by_user_id%2Chs_user_ids_of_all_notification_followers%2Chs_user_ids_of_all_notification_unfollowers%2Chs_user_ids_of_all_owners%2Chs_was_imported%2Chubspot_owner_assigneddate%2Chubspot_owner_id%2Chubspot_team_id%2Chs_all_owner_ids%2Chs_all_team_ids%2Chs_all_accessible_team_ids returned 403: Forbidden. This is most likely due to insufficient permissions on the credentials in use. Try to grant required permissions/scopes or re-authenticate. Please visit https://docs.airbyte.com/integrations/sources/hubspot to learn more.  Invalid permissions for engagements_emails. Please ensure the all scopes are authorized for. Missing scopes ['sales-email-read']
Syncing stream contacts 0:06:39.402617
2024-06-26 03:33:58 source > Reading contacts associations of meetings
Syncing stream contacts 0:06:39.402617
2024-06-26 03:33:58 source > Reading contacts associations of notes
Syncing stream contacts 0:06:39.402617
2024-06-26 03:33:58 source > Reading contacts associations of tasks
2024-06-26 03:34:05 source > Reading contacts associations of tasks
2024-06-26 03:34:12 source > Reading contacts associations of tasks
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617

Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
2024-06-26 04:08:06 source > Reading contacts associations of ticket
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
Syncing stream contacts 0:06:39.402617
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _contacts. Error messages: [$.companies[1]: integer found, but [null, string] is required, $.companies[0]: integer found, but [null, string] is required]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _engagements_meetings. Error messages: [$.contacts[0]: integer found, but [null, string] is required]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _engagements_notes. Error messages: [$.tickets[0]: integer found, but [null, string] is required]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _tickets. Error messages: [$.companies[1]: integer found, but [null, string] is required, $.deals[0]: integer found, but [null, string] is required, $.companies[0]: integer found, but [null, string] is required]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _companies. Error messages: [$.contacts[160]: integer found, string expected, $.contacts[78]: integer found, string expected, $.contacts[168]: integer found, string expected, $.contacts[76]: integer found, string expected, $.contacts[74]: integer found, string expected, $.contacts[166]: integer found, string expected, $.contacts[70]: integer found, string expected, $.contacts[164]: integer found, string expected, $.contacts[162]: integer found, string expected, $.contacts[72]: integer found, string expected, $.contacts[159]: integer found, string expected, $.contacts[179]: integer found, string expected, $.contacts[177]: integer found, string expected, $.contacts[175]: integer found, string expected, $.contacts[90]: integer found, string expected, $.contacts[173]: integer found, string expected, $.contacts[171]: integer found, string expected, $.contacts[85]: integer found, string expected, $.contacts[87]: integer found, string expected, $.contacts[81]: integer found, string expected, $.contacts[83]: integer found, string expected, $.contacts[56]: integer found, string expected, $.contacts[54]: integer found, string expected, $.contacts[52]: integer found, string expected, $.contacts[50]: integer found, string expected, $.contacts[89]: integer found, string expected, $.contacts[139]: integer found, string expected, $.contacts[2]: integer found, string expected, $.contacts[43]: integer found, string expected, $.contacts[47]: integer found, string expected, $.contacts[142]: integer found, string expected, $.contacts[199]: integer found, string expected, $.contacts[195]: integer found, string expected, $.contacts[201]: integer found, string expected, $.contacts[146]: integer found, string expected, $.contacts[191]: integer found, string expected, $.contacts[126]: integer found, string expected, $.contacts[98]: integer found, string expected, $.contacts[94]: integer found, string expected, $.contacts[63]: integer found, string expected, $.contacts[122]: integer found, string expected, $.contacts[13]: integer found, string expected, $.contacts[209]: integer found, string expected, $.contacts[212]: integer found, string expected, $.contacts[17]: integer found, string expected, $.contacts[157]: integer found, string expected, $.contacts[205]: integer found, string expected, $.contacts[67]: integer found, string expected, $.contacts[119]: integer found, string expected, $.contacts[180]: integer found, string expected, $.contacts[115]: integer found, string expected, $.contacts[103]: integer found, string expected, $.contacts[153]: integer found, string expected, $.contacts[32]: integer found, string expected, $.contacts[188]: integer found, string expected, $.contacts[36]: integer found, string expected, $.contacts[184]: integer found, string expected, $.contacts[228]: integer found, string expected, $.contacts[23]: integer found, string expected, $.contacts[26]: integer found, string expected, $.contacts[29]: integer found, string expected, $.contacts[223]: integer found, string expected, $.contacts[224]: integer found, string expected, $.contacts[221]: integer found, string expected, $.contacts[28]: integer found, string expected, $.contacts[109]: integer found, string expected, $.contacts[104]: integer found, string expected, $.contacts[114]: integer found, string expected, $.contacts[20]: integer found, string expected, $.contacts[21]: integer found, string expected, $.contacts[106]: integer found, string expected, $.contacts[112]: integer found, string expected, $.contacts[218]: integer found, string expected, $.contacts[215]: integer found, string expected, $.contacts[231]: integer found, string expected, $.contacts[232]: integer found, string expected, $.contacts[125]: integer found, string expected, $.contacts[10]: integer found, string expected, $.contacts[123]: integer found, string expected, $.contacts[12]: integer found, string expected, $.contacts[213]: integer found, string expected, $.contacts[18]: integer found, string expected, $.contacts[204]: integer found, string expected, $.contacts[31]: integer found, string expected, $.contacts[229]: integer found, string expected, $.contacts[8]: integer found, string expected, $.contacts[37]: integer found, string expected, $.contacts[131]: integer found, string expected, $.contacts[133]: integer found, string expected, $.contacts[39]: integer found, string expected, $.contacts[0]: integer found, string expected, $.contacts[58]: integer found, string expected, $.contacts[45]: integer found, string expected, $.contacts[42]: integer found, string expected, $.contacts[202]: integer found, string expected, $.contacts[144]: integer found, string expected, $.contacts[147]: integer found, string expected, $.contacts[198]: integer found, string expected, $.contacts[96]: integer found, string expected, $.contacts[128]: integer found, string expected, $.contacts[99]: integer found, string expected, $.contacts[193]: integer found, string expected, $.contacts[190]: integer found, string expected, $.contacts[120]: integer found, string expected, $.contacts[91]: integer found, string expected, $.contacts[61]: integer found, string expected, $.contacts[174]: integer found, string expected, $.contacts[15]: integer found, string expected, $.contacts[207]: integer found, string expected, $.contacts[155]: integer found, string expected, $.contacts[69]: integer found, string expected, $.contacts[210]: integer found, string expected, $.contacts[88]: integer found, string expected, $.contacts[117]: integer found, string expected, $.contacts[182]: integer found, string expected, $.contacts[101]: integer found, string expected, $.contacts[34]: integer found, string expected, $.contacts[5]: integer found, string expected, $.contacts[53]: integer found, string expected, $.contacts[226]: integer found, string expected, $.contacts[136]: integer found, string expected, $.contacts[161]: integer found, string expected, $.contacts[79]: integer found, string expected, $.contacts[169]: integer found, string expected, $.contacts[77]: integer found, string expected, $.contacts[167]: integer found, string expected, $.contacts[75]: integer found, string expected, $.contacts[80]: integer found, string expected, $.contacts[71]: integer found, string expected, $.contacts[165]: integer found, string expected, $.contacts[73]: integer found, string expected, $.contacts[163]: integer found, string expected, $.contacts[66]: integer found, string expected, $.contacts[64]: integer found, string expected, $.contacts[60]: integer found, string expected, $.contacts[62]: integer found, string expected, $.contacts[149]: integer found, string expected, $.contacts[154]: integer found, string expected, $.contacts[156]: integer found, string expected, $.contacts[68]: integer found, string expected, $.contacts[158]: integer found, string expected, $.contacts[181]: integer found, string expected, $.contacts[183]: integer found, string expected, $.contacts[152]: integer found, string expected, $.contacts[150]: integer found, string expected, $.contacts[189]: integer found, string expected, $.contacts[187]: integer found, string expected, $.contacts[185]: integer found, string expected, $.contacts[138]: integer found, string expected, $.contacts[3]: integer found, string expected, $.contacts[59]: integer found, string expected, $.contacts[141]: integer found, string expected, $.contacts[48]: integer found, string expected, $.contacts[40]: integer found, string expected, $.contacts[145]: integer found, string expected, $.contacts[196]: integer found, string expected, $.contacts[200]: integer found, string expected, $.contacts[97]: integer found, string expected, $.contacts[192]: integer found, string expected, $.contacts[127]: integer found, string expected, $.contacts[93]: integer found, string expected, $.contacts[44]: integer found, string expected, $.contacts[176]: integer found, string expected, $.contacts[172]: integer found, string expected, $.contacts[86]: integer found, string expected, $.contacts[82]: integer found, string expected, $.contacts[7]: integer found, string expected, $.contacts[55]: integer found, string expected, $.contacts[130]: integer found, string expected, $.contacts[134]: integer found, string expected, $.contacts[51]: integer found, string expected, $.contacts[24]: integer found, string expected, $.contacts[220]: integer found, string expected, $.contacts[25]: integer found, string expected, $.contacts[27]: integer found, string expected, $.contacts[222]: integer found, string expected, $.contacts[108]: integer found, string expected, $.contacts[110]: integer found, string expected, $.contacts[113]: integer found, string expected, $.contacts[105]: integer found, string expected, $.contacts[22]: integer found, string expected, $.contacts[111]: integer found, string expected, $.contacts[107]: integer found, string expected, $.contacts[219]: integer found, string expected, $.contacts[214]: integer found, string expected, $.contacts[216]: integer found, string expected, $.contacts[217]: integer found, string expected, $.contacts[230]: integer found, string expected, $.contacts[121]: integer found, string expected, $.contacts[14]: integer found, string expected, $.contacts[208]: integer found, string expected, $.contacts[16]: integer found, string expected, $.contacts[206]: integer found, string expected, $.contacts[211]: integer found, string expected, $.contacts[118]: integer found, string expected, $.contacts[116]: integer found, string expected, $.contacts[102]: integer found, string expected, $.contacts[100]: integer found, string expected, $.contacts[33]: integer found, string expected, $.contacts[35]: integer found, string expected, $.contacts[227]: integer found, string expected, $.contacts[6]: integer found, string expected, $.contacts[225]: integer found, string expected, $.contacts[135]: integer found, string expected, $.contacts[137]: integer found, string expected, $.contacts[1]: integer found, string expected, $.contacts[4]: integer found, string expected, $.contacts[57]: integer found, string expected, $.contacts[46]: integer found, string expected, $.contacts[41]: integer found, string expected, $.contacts[143]: integer found, string expected, $.contacts[49]: integer found, string expected, $.contacts[197]: integer found, string expected, $.contacts[129]: integer found, string expected, $.contacts[194]: integer found, string expected, $.contacts[92]: integer found, string expected, $.contacts[95]: integer found, string expected, $.contacts[140]: integer found, string expected, $.contacts[65]: integer found, string expected, $.contacts[178]: integer found, string expected, $.contacts[124]: integer found, string expected, $.contacts[11]: integer found, string expected, $.contacts[170]: integer found, string expected, $.contacts[148]: integer found, string expected, $.contacts[30]: integer found, string expected, $.contacts[19]: integer found, string expected, $.contacts[203]: integer found, string expected, $.contacts[84]: integer found, string expected, $.contacts[151]: integer found, string expected, $.contacts[132]: integer found, string expected, $.contacts[186]: integer found, string expected, $.contacts[38]: integer found, string expected, $.contacts[9]: integer found, string expected]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _engagements_calls. Error messages: [$.contacts[0]: integer found, but [null, string] is required]
2024-06-26 04:08:11 replication-orchestrator > Schema validation errors found for stream _engagements_tasks. Error messages: [$.contacts[5]: integer found, but [null, string] is required, $.contacts[1]: integer found, but [null, string] is required, $.contacts[4]: integer found, but [null, string] is required, $.contacts[2]: integer found, but [null, string] is required, $.contacts[0]: integer found, but [null, string] is required, $.contacts[3]: integer found, but [null, string] is required]

  "status" : "completed",
  "recordsSynced" : 70778,
  "bytesSynced" : 1407096897,
  "startTime" : 1719371336010,
  "endTime" : 1719374926477,
  "totalStats" : {
    "bytesCommitted" : 1407096897,
    "bytesEmitted" : 1407096897,
    "destinationStateMessagesEmitted" : 33,
    "destinationWriteEndTime" : 1719374926371,
    "destinationWriteStartTime" : 1719371336110,
    "meanSecondsBeforeSourceStateMessageEmitted" : 42,
    "maxSecondsBeforeSourceStateMessageEmitted" : 14,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 108,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 70,
    "recordsEmitted" : 70778,
    "recordsCommitted" : 70778,
    "replicationEndTime" : 1719374926474,
    "replicationStartTime" : 1719371336010,
    "sourceReadEndTime" : 1719374891392,
    "sourceReadStartTime" : 1719371336111,
    "sourceStateMessagesEmitted" : 34
  },
  "streamStats" : [ {
    "streamName" : "contacts",
    "stats" : {
      "bytesCommitted" : 439158807,
      "bytesEmitted" : 439158807,
      "recordsEmitted" : 6971,
      "recordsCommitted" : 6971
    }
  }],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 23931251730,
      "executionCount" : 70869,
      "avgExecTimeInNanos" : 337682.9323117301
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 2000945715191,
      "executionCount" : 110052,
      "avgExecTimeInNanos" : 1.818182055020354E7
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 243316636,
      "executionCount" : 33,
      "avgExecTimeInNanos" : 7373231.393939394
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 1648076224856,
      "executionCount" : 70812,
      "avgExecTimeInNanos" : 2.3273968040106196E7
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 3578986673112,
      "executionCount" : 372501,
      "avgExecTimeInNanos" : 9607992.121127192
    }
  }
}
2024-06-26 04:08:46 replication-orchestrator > failures: [ ]
2024-06-26 04:08:46 replication-orchestrator > Returning output...
2024-06-26 04:08:46 replication-orchestrator > 
2024-06-26 04:08:46 replication-orchestrator > ----- END REPLICATION -----
2024-06-26 04:08:46 replication-orchestrator > 
2024-06-26 04:08:47 replication-orchestrator > Writing async status SUCCEEDED for KubePodInfo[namespace=data-airbyte, name=orchestrator-repl-job-4212-attempt-0, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:0.62.4, pullPolicy=IfNotPresent]]...
2024-06-26 04:08:47 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):296 - Waiting for Segment analytic client to flush enqueued messages...
2024-06-26 04:08:47 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):308 - Segment analytic client flush complete.
2024-06-26 04:08:47 INFO i.a.a.SegmentAnalyticsClient(close):244 - Segment analytics client closed.  No new events will be accepted.
2024-06-26 04:08:51 platform > State Store reports orchestrator pod orchestrator-repl-job-4212-attempt-0 succeeded

Contribute

marcosmarxm commented 1 week ago

@artem1205 and @airbytehq/critical-connectors can someone take a look in this?

Sorry the lack of updates here @ASmith12345

bazarnov commented 1 week ago

@marcosmarxm Just FYI, no need to tag for critical-sources specifically. We moved to the @airbytehq/dev-python instead.

I'll take a look.

bazarnov commented 1 week ago

@ASmith12345 Could you please share your Airbyte Cloud connection link, so we can investigate and reproduce the issue?