search

DataReply / kafka-connect-mongodb

Apache License 2.0
129 stars 61 forks source link

Use Mongo as source and sink #14

Open mongodbuser opened 8 years ago

mongodbuser commented 8 years ago

I have not been able to get this working with MongoDB as source as well as sink. These servers are not a part of the replica set and there is a need to replay all events as it is.

Questions:

  1. I assume scheme registry is not applicable here. Please confirm
  2. What is the definition of schema.name - Can any values be used here?

Following are the property files:

Source

name=mongodb-source-connector connector.class=org.apache.kafka.connect.mongodb.MongodbSourceConnector tasks.max=1 host=10.6.156.188 port=27017 batch.size=100 topic.prefix=testconn schema.name=mongodbschema databases=PocDB.myCollection

Sink

name=mongodb-sink-connector connector.class=org.apache.kafka.connect.mongodb.MongodbSinkConnector tasks.max=1 host=10.6.156.189 port=27017 bulk.size=100 mongodb.database=PocDB mongodb.collections=myCollection topics=testconn_PocDB_myCollection

Any help in this regard is appreciated.

Thanks

xuxu0728 commented 7 years ago

hi,did you run it!When I run the mongodb as source,there is no work.Ａｎｄ the error which zookeeper output like this



INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:13:53,344] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/owners/test1 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-46190/owners/test1 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:13:53,347] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-46190/owners (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:14:53,203] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:setData cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/offsets/test1/0 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-46190/offsets/test1/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:14:53,206] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/offsets Error:KeeperErrorCode = NoNode for /consumers/console-consumer-46190/offsets (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:14:53,214] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:setData cxid:0x2b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/offsets/test1/1 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-46190/offsets/test1/1 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-08-14 15:14:53,216] INFO Got user-level KeeperException when processing sessionid:0x15ddf911b8a000b type:create cxid:0x2c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/console-consumer-46190/offsets/test1 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-46190/offsets/test1 (org.apache.zookeeper.server.PrepRequestProcessor)

Could you help me?
zoeminghong commented 6 years ago

@mongodbuser can you show connect-avro-standalone.properties's code? About the schema.name, you see #19