Source CockroachDB: fails at >10M records

[eembees]

Environment

• Airbyte version: 0.39.7-alpha and 0.39.5-alpha
• OS Version / Instance: Ubuntu 18.04.5 LTS
• Deployment: Docker-Compose (directly from airbyte repo)
• Source Connector and version: (if applicable example Salesforce 0.2.3) CockroachDB 0.1.12
• Destination Connector and version: (if applicable example Postgres 0.3.3) Postgres 0.3.20
• Step where error happened: Sync job

Current Behavior

When importing a 18.1Gb (57M Records) table from CRDB using the Cockroach connector, the sync job fails. Before failing, the adapter fills all available RAM before the container shuts down: (presumably an OOM).

This does not happen on smaller tables with about 1M records, with the same source and destination:

Expected Behavior

Airbyte should

1. not throw OOM when transferring the data
2. allow for manual setting of batch size if dynamic batch size fails

Logs

logs-5.txt

Steps to Reproduce

1. Spin up airbyte, postgres and crdb (open source containers) as described in respective docs
2. Populate CRDB
3. Connect CRDB cluster to airbyte
4. attempt to transfer a large table (20gB or 60M records)

Are you willing to submit a PR?

I have no experience with Java, but any Python / Go code we can work with.

[marcosmarxm]

Discussion in Discourse herE: https://discuss.airbyte.io/t/cockroachdb-source-connector-failures-at-10m-records/1261

[eembees]

I have found this issue, which seems to have the same problem: https://github.com/airbytehq/airbyte/issues/25276 I expect both these issues to be fixed by adding a LIMIT clause as suggested by @lukeasrodgers here https://github.com/airbytehq/airbyte/blob/f23b3ad88ed56c605ea1ccad857e287b7a38dbe0/airbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java#L113 .

[tomerpeled]

Any updates on this issue?