Source Connector connected but not fetching data from mongodb

[AAbhardwaj]

Hi Source Connector connected but not fetching data from mongodb. Please advice.

[2017-12-05 16:59:58,009] INFO Cluster created with settings {hosts=[127.0.0.1:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500} (org.mongodb.driver.cluster:71) [2017-12-05 16:59:58,016] INFO Cluster description not yet available. Waiting for 30000 ms before timing out (org.mongodb.driver.cluster:71) [2017-12-05 16:59:58,019] INFO Opened connection [connectionId{localValue:3, serverValue:282}] to 127.0.0.1:27017 (org.mongodb.driver.connection:71) [2017-12-05 16:59:58,019] INFO Monitor thread successfully connected to server with description ServerDescription{address=127.0.0.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 10]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=280047} (org.mongodb.driver.cluster:71) [2017-12-05 16:59:58,020] INFO WorkerSourceTask{id=mongodb-source-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:158) [2017-12-05 16:59:58,024] INFO Opened connection [connectionId{localValue:4, serverValue:283}] to 127.0.0.1:27017 (org.mongodb.driver.connection:71) [2017-12-05 16:59:58,024] INFO Opened connection [connectionId{localValue:5, serverValue:284}] to 127.0.0.1:27017 (org.mongodb.driver.connection:71) [2017-12-05 16:59:58,024] INFO Opened connection [connectionId{localValue:6, serverValue:285}] to 127.0.0.1:27017 (org.mongodb.driver.connection:71) [2017-12-05 17:00:57,903] INFO WorkerSourceTask{id=mongodb-source-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:306) [2017-12-05 17:00:57,904] INFO WorkerSourceTask{id=mongodb-source-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:323)

Best regards,

[AAbhardwaj]

$ curl localhost:8083/connectors/mongodb-source-connector/tasks [ { "id": { "connector":"mongodb-source-connector", "task":0 }, "config":{ "databases":"db.movies,mydb.ratings,db.usersAndMovies", "topic.prefix":"abhishek", "task.class":"org.apache.kafka.connect.mongodb.MongodbSourceTask", "schema.name":"test", "batch.size":"100", "converter.class":"org.apache.kafka.connect.mongodb.converter.JsonStructConverter" ,"uri":"mongodb://127.0.0.1:27017" } } ]

[AAbhardwaj]

$curl localhost:8083/connectors/mongodb-source-connector/status { "name":"mongodb-source-connector", "connector":{ "state":"RUNNING", "worker_id":"1xx.2x.xx.xx:8083" }, "tasks": [ { "state":"RUNNING", "id":0, "worker_id":"1xx.2x.xx.xx:8083" } ], "type":"source" }

[AAbhardwaj]

I do not get any messages in my kafka topic. Any idea?

[ruileal]

As the documentation of this connector says: "When the connector is run as a Source Connector, it reads data from Mongodb oplog and publishes it on Kafka". See: https://docs.mongodb.com/manual/core/replica-set-oplog/

Check that your mongo DB has the collection "local.oplog.rs". If not, it's because is not set up to be a replica set. As soon as you set it, that collection is created in your mongo and the connector is able to start creating and sending the messages to kafka.

[nvitucci]

As @ruileal said, MongoDB has to be configured as a replica set. In short, to make it work:

• stop MongoDB;
• edit the configuration file (usually in /etc/mongod.conf), uncomment the replication line and add a replSetName: rs line;
• start MongoDB again;
• execute rs.initiate() from its command line.

Then, you should start seeing the oplog messages.

[sangeesivakumar]

@system103:~$ rs.initiate()

I am not getting oplog messages

[anupkumarsharma]

Does mongo have replication set? Check this - https://docs.mongodb.com/manual/core/replica-set-oplog/. Also, it might be easy to just clone the source and run test case- testInsertWithNullOffsets by setting your connection string. You have to comment a bunch of things, but it's easier to debug issues that way.