Destination ClickHouse: out of memory with Incremental - Deduped History model

[yuxh]

Environment

• Airbyte version: 0.39.41-alpha
• OS Version / Instance: centos
• Deployment: Docker
• Source Connector and version: mongodb
• Destination Connector and version:clickhouse
• Step where error happened: Incremental - Deduped History Sync job

Current Behavior

Tell us what happens. "DB::Exception: Memory limit (for query) exceeded: .." , details in the log below. It seems too many memories (even 32 GB not enough) needed to generate _scd, although the data in mongodb has only millions of records.

Expected Behavior

Tell us what should happen. Any chance to reduce memory, like generate in batch?

Logs

logs-21.txt

[IzioDev]

Potentially related issue : #9553

[zczhuohuo]

I came across the same issue too. Clickhouse normalization consume way too much memory.