The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
I’ve deployed Airbyte successfully to our K8S cluster via the Helm chart, We were able to create few connections from our atlas mongo env's to the postgresql.
But we are facing issue with one connection(uat3 to postgresql failed with error: Failed to fetch schema)
Error
Log4j2Appender says: 2022-05-24 17:46:00 [32mINFO[m i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='input_state.json'}
2022-05-24 17:46:00 [1;31mERROR[m i.a.c.i.LineGobbler(voidCall):82 - Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 292 (QueryExceededMemoryLimitNoDiskUseAllowed): 'Error in $cursor stage :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.' on server xxxxxxxxxxxxxx.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1653414360, "i": 15}}, "ok": 0.0, "errmsg": "Error in $cursor stage :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.", "code": 292, "codeName": "**QueryExceededMemoryLimitNoDiskUseAllowed**", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1653414360, "i": 15}}, "signature": {"hash": {"$binary": {"base64": "4r13EluCoBSqz6gDFm03r0AuY0c=", "subType": "00"}}, "keyId": 7064253550083178498}}}
Environment
Deployment: Kubernetes (AWS EKS v1.21) Helm Chart v0.3.2 Airbyte Version: 0.35.12-alpha Source name/version: Gitlab v0.1.5
Step: Connection Creation
I’ve deployed Airbyte successfully to our K8S cluster via the Helm chart, We were able to create few connections from our atlas mongo env's to the postgresql.
But we are facing issue with one connection(uat3 to postgresql failed with error: Failed to fetch schema)
Error