The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Step where error happened: Incremental - Deduped History Sync job
Current Behavior
Tell us what happens.
"DB::Exception: Memory limit (for query) exceeded: .." , details in the log below.
It seems too many memories (even 32 GB not enough) needed to generate _scd, although the data in mongodb has only millions of records.
Expected Behavior
Tell us what should happen.
Any chance to reduce memory, like generate in batch?
Environment
Current Behavior
Tell us what happens. "DB::Exception: Memory limit (for query) exceeded: .." , details in the log below. It seems too many memories (even 32 GB not enough) needed to generate _scd, although the data in mongodb has only millions of records.
Expected Behavior
Tell us what should happen. Any chance to reduce memory, like generate in batch?
Logs
logs-21.txt