search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.13k stars 4.12k forks source link

Source MongoDd: v2 v1.0.1 certified version cannot work with MongoDB Atlas sharded cluster #31467

Closed gititdom closed 4 months ago

gititdom commented 1 year ago

Connector Name

source-MongoDb

Connector Version

1.0.1

What step the error happened?

Configuring a new connector

Relevant information

The prior MongoDb community version source connector was able to work with the MongoDB Atlas Sharded Cluster. However, the sharded cluster no longer works with the new certified version. It throws the error "Target MongoDB instance is not a replica set cluster." during connection configuration.

Relevant log output

2023-10-16 18:15:54 INFO i.a.w.t.TemporalAttemptExecution(get):124 - Docker volume job log path: /tmp/workspace/67a7fa0a-a618-4a02-ac36-9e6555f739f4/0/logs.log
2023-10-16 18:15:54 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.30
2023-10-16 18:15:54 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-10-16 18:15:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-10-16 18:15:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-10-16 18:15:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-10-16 18:15:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-10-16 18:15:54 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-10-16 18:15:54 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-10-16 18:15:54 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-10-16 18:15:54 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-10-16 18:15:54 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-mongodb-v2:1.0.1 exists...
2023-10-16 18:15:54 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-mongodb-v2:1.0.1 was found locally.
2023-10-16 18:15:54 INFO i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = source-mongodb-v2-check-67a7fa0a-a618-4a02-ac36-9e6555f739f4-0-udthe with resources io.airbyte.config.ResourceRequirements@2c98fc1b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-10-16 18:15:54 INFO i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/67a7fa0a-a618-4a02-ac36-9e6555f739f4/0 --log-driver none --name source-mongodb-v2-check-67a7fa0a-a618-4a02-ac36-9e6555f739f4-0-udthe --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-mongodb-v2:1.0.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.30 -e WORKER_JOB_ID=67a7fa0a-a618-4a02-ac36-9e6555f739f4 airbyte/source-mongodb-v2:1.0.1 check --config source_config.json
2023-10-16 18:15:54 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO i.a.i.s.m.MongoDbSource(main):44 - starting source: class io.airbyte.integrations.source.mongodb.MongoDbSource
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.source.mongodb.MongoDbSource
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 WARN c.m.i.d.l.SLF4JLogger(warn):86 - Connection string contains unsupported option 'provider'.
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - MongoClient with metadata {"driver": {"name": "mongo-java-driver|sync|Airbyte", "version": "4.10.2"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.19.0-25-cloud-amd64"}, "platform": "Java/Amazon.com Inc./17.0.8.1+8-LTS"} created with settings MongoClientSettings{readPreference=ReadPreference{name=secondaryPreferred, hedgeOptions=null}, writeConcern=WriteConcern{w=majority?retryWrites=false, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='airbyte', source='admin', password=<hidden>, mechanismProperties=<hidden>}, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@62ddd21b, com.mongodb.Jep395RecordCodecProvider@16c3ca31, com.mongodb.KotlinCodecProvider@2d195ee4]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[127.0.0.1:27017], srvHost=xxxxxx-pl-0.XXXXX.mongodb.net, srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - Cluster description not yet available. Waiting for 30000 ms before timing out
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-034-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-045-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:57 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-001-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:57 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-000-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-012-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Adding discovered server pl-00-023-us-west4-gcp.XXXXX.mongodb.net:27016 to client view of cluster
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-023-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=649201044}
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-012-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=650074207}
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-001-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=651954767}
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-034-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=659585077}
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-045-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=657596251}
2023-10-16 18:15:58 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:58 INFO c.m.i.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=pl-00-000-us-west4-gcp.XXXXX.mongodb.net:27016, type=SHARD_ROUTER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=655316907}
2023-10-16 18:15:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:59 INFO c.m.i.d.l.SLF4JLogger(info):71 - Discovered cluster type of SHARDED
2023-10-16 18:15:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:59 ERROR i.a.i.s.m.MongoDbSource(check):68 - Target MongoDB instance is not a replica set cluster.
2023-10-16 18:15:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:59 INFO i.a.c.i.b.IntegrationRunner(runInternal):228 - Completed integration: io.airbyte.integrations.source.mongodb.MongoDbSource
2023-10-16 18:15:59 INFO i.a.w.i.VersionedAirbyteStreamFactory(toAirbyteMessage):364 - 2023-10-16 18:15:59 INFO i.a.i.s.m.MongoDbSource(main):46 - completed source: class io.airbyte.integrations.source.mongodb.MongoDbSource
2023-10-16 18:15:59 INFO i.a.w.g.DefaultCheckConnectionWorker(run):117 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@36417223[status=failed,message=Target MongoDB instance is not a replica set cluster.,additionalProperties={}]
2023-10-16 18:15:59 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-10-16 18:15:59 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-10-16 18:15:59 INFO i.a.c.i.LineGobbler(voidCall):149 -

Contribute

evantahler commented 4 months ago

Closing as stale - please re-open if the issue persists