airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.52k stars 4k forks source link

Set up connection failed to fetch schema from Mongodb #13185

Closed pratyushareddym closed 2 years ago

pratyushareddym commented 2 years ago

Environment

Deployment: Kubernetes (AWS EKS v1.21) Helm Chart v0.3.2 Airbyte Version: 0.35.12-alpha Source name/version: Gitlab v0.1.5

Step: Connection Creation

I’ve deployed Airbyte successfully to our K8S cluster via the Helm chart, We were able to create few connections from our atlas mongo env's to the postgresql.

But we are facing issue with one connection(uat3 to postgresql failed with error: Failed to fetch schema)

Error

Log4j2Appender says: 2022-05-24 17:46:00 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='input_state.json'}
2022-05-24 17:46:00 ERROR i.a.c.i.LineGobbler(voidCall):82 - Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 292 (QueryExceededMemoryLimitNoDiskUseAllowed): 'Error in $cursor stage :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.' on server xxxxxxxxxxxxxx.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1653414360, "i": 15}}, "ok": 0.0, "errmsg": "Error in $cursor stage :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.", "code": 292, "codeName": "**QueryExceededMemoryLimitNoDiskUseAllowed**", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1653414360, "i": 15}}, "signature": {"hash": {"$binary": {"base64": "4r13EluCoBSqz6gDFm03r0AuY0c=", "subType": "00"}}, "keyId": 7064253550083178498}}}
marcosmarxm commented 2 years ago

It's not clear the source is Gitlab or Mongodb? Please upload the complete log file as describe in the issue template.

marcosmarxm commented 2 years ago

Close due inactivity and doesn't have complete information.