Closed nVitius closed 8 years ago
Hello!
I'm going to need more information from you. How are you running mongo-connector?
Are there any errors, and have you looked at the mongo-connector.log file or the elastic logs?
Based on the title of this issue, I'd like to ask a question not really connected to OPs problem.
Can mongo-connector be configured to ignore updates and deletes? I want to use it only to sync inserts from mongodb to elasticsearch (elastic2-doc-manager).
@mkozjak hi, it would be better if you filed a separate issue because your question is not really related to this issue.
Sure.
I'm running the connector in a docker container like so:
mongo-connector \
--batch-size 100 \
--auto-commit-interval=5 \
-m mongo \
--namespace-set voter.voter \
-t elasticsearch \
-d elastic2_doc_manager -v --stdout \
--oplog-ts /data/oplog.timestamp
Given this entity:
{
"_id": "5702bb014cf4cb1b0070dedb",
"created": "2016-04-04T19:05:37.180Z",
"modified": "2016-04-04T19:05:37.190Z",
"vitals": {
"citizen": true,
"last": "Daniels",
"first": "Bethany",
"middle": "C",
"sex": "Female",
"dob": "1976-03-14T08:00:00.000Z",
"created": "2016-04-04T19:05:37.166Z"
},
"identification": [],
"contact": {
"residence": [
{
"address": "284 Market",
"city": "Newark",
"state": "NJ",
"zipcode": "07102",
"created": "2016-04-04T19:05:37.167Z",
"addressId": 0
}
],
"mailing": [
{
"address": "284 Market",
"address2": "",
"city": "Newark",
"state": "NJ",
"zipcode": "07102",
"created": "2016-04-04T19:05:37.167Z",
"addressId": 0
}
],
"inactiveConfirmation": [],
"phone": [],
"email": [
{
"email": "test@test.com",
"created": "2016-04-04T19:05:37.167Z"
}
]
},
"status": [],
"voterId": "EJfQ5isAe"
}
I change the last name from Daniels
to Daniel
. This is the output I get from the logs:
2016-04-04 19:21:14,197 [DEBUG] mongo_connector.oplog_manager:188 - OplogThread: Cursor is still alive and thread is still running.
2016-04-04 19:21:14,814 [DEBUG] mongo_connector.oplog_manager:194 - OplogThread: Iterating through cursor, document number in this cursor is 0
2016-04-04 19:21:14,815 [DEBUG] mongo_connector.oplog_manager:239 - OplogThread: Operation for this entry is u
2016-04-04 19:21:14,815 [DEBUG] urllib3.util.retry:156 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-04-04 19:21:14,818 [DEBUG] urllib3.connectionpool:386 - "POST /_refresh HTTP/1.1" 200 51
2016-04-04 19:21:14,818 [INFO] elasticsearch:63 - POST http://elasticsearch:9200/_refresh [status:200 request:0.003s]
2016-04-04 19:21:14,818 [DEBUG] elasticsearch:65 - > None
2016-04-04 19:21:14,818 [DEBUG] elasticsearch:66 - < {"_shards":{"total":26,"successful":13,"failed":0}}
2016-04-04 19:21:14,820 [DEBUG] urllib3.util.retry:156 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-04-04 19:21:14,845 [DEBUG] urllib3.connectionpool:386 - "GET /voter/voter/5702bb014cf4cb1b0070dedb HTTP/1.1" 200 870
2016-04-04 19:21:14,846 [INFO] elasticsearch:63 - GET http://elasticsearch:9200/voter/voter/5702bb014cf4cb1b0070dedb [status:200 request:0.025s]
2016-04-04 19:21:14,846 [DEBUG] elasticsearch:65 - > None
2016-04-04 19:21:14,846 [DEBUG] elasticsearch:66 - < {"_index":"voter","_type":"voter","_id":"5702bb014cf4cb1b0070dedb","_version":1,"found":true,"_source":{"status": [], "created": "2016-04-04T19:05:37.180000", "modified": "2016-04-04T19:05:37.190000", "voterId": "EJfQ5isAe", "contact": {"residence": [{"city": "Newark", "addressId": 0, "created": "2016-04-04T19:05:37.167000", "zipcode": "07102", "state": "NJ", "address": "284 Market"}], "inactiveConfirmation": [], "mailing": [{"city": "Newark", "addressId": 0, "created": "2016-04-04T19:05:37.167000", "address2": "", "zipcode": "07102", "state": "NJ", "address": "284 Market"}], "phone": [], "email": [{"email": "test@test.com", "created": "2016-04-04T19:05:37.167000"}]}, "identification": [], "vitals": {"last": "Daniels", "created": "2016-04-04T19:05:37.166000", "dob": "1976-03-14T08:00:00", "sex": "Female", "middle": "C", "citizen": true, "first": "Bethany"}}}
2016-04-04 19:21:14,852 [DEBUG] urllib3.util.retry:156 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-04-04 19:21:14,905 [DEBUG] urllib3.connectionpool:386 - "PUT /voter/voter/5702bb014cf4cb1b0070dedb?refresh=false HTTP/1.1" 200 144
2016-04-04 19:21:14,905 [INFO] elasticsearch:63 - PUT http://elasticsearch:9200/voter/voter/5702bb014cf4cb1b0070dedb?refresh=false [status:200 request:0.054s]
2016-04-04 19:21:14,906 [DEBUG] elasticsearch:65 - > {"status": [], "created": "2016-04-04T19:05:37.180Z", "modified": "2016-04-04T19:05:37.190Z", "voterId": "EJfQ5isAe", "contact": {"residence": [{"city": "Newark", "addressId": 0, "created": "2016-04-04T19:05:37.167Z", "zipcode": "07102", "state": "NJ", "address": "284 Market"}], "inactiveConfirmation": [], "mailing": [{"city": "Newark", "addressId": 0, "created": "2016-04-04T19:05:37.167Z", "address2": "", "zipcode": "07102", "state": "NJ", "address": "284 Market"}], "phone": [], "email": [{"email": "test@test.com", "created": "2016-04-04T19:05:37.167Z"}]}, "identification": [], "vitals": {"last": "Daniel", "created": "2016-04-04T19:05:37.166Z", "dob": "1976-03-14T08:00:00.000Z", "sex": "Female", "middle": "C", "citizen": true, "first": "Bethany"}}
2016-04-04 19:21:14,906 [DEBUG] elasticsearch:66 - < {"_index":"voter","_type":"voter","_id":"5702bb014cf4cb1b0070dedb","_version":2,"_shards":{"total":2,"successful":1,"failed":0},"created":false}
2016-04-04 19:21:14,906 [DEBUG] urllib3.util.retry:156 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-04-04 19:21:14,926 [DEBUG] urllib3.connectionpool:386 - "PUT /mongodb_meta/mongodb_meta/5702bb014cf4cb1b0070dedb?refresh=false HTTP/1.1" 200 158
2016-04-04 19:21:14,926 [INFO] elasticsearch:63 - PUT http://elasticsearch:9200/mongodb_meta/mongodb_meta/5702bb014cf4cb1b0070dedb?refresh=false [status:200 request:0.020s]
2016-04-04 19:21:14,926 [DEBUG] elasticsearch:65 - > {"ns": "voter.voter", "_ts": 6269783268606869505}
2016-04-04 19:21:14,926 [DEBUG] elasticsearch:66 - < {"_index":"mongodb_meta","_type":"mongodb_meta","_id":"5702bb014cf4cb1b0070dedb","_version":2,"_shards":{"total":2,"successful":1,"failed":0},"created":false}
2016-04-04 19:21:14,928 [DEBUG] mongo_connector.oplog_manager:294 - OplogThread: Doc is processed.
2016-04-04 19:21:15,930 [DEBUG] mongo_connector.oplog_manager:306 - OplogThread: updating checkpoint afterprocessing new oplog entries
2016-04-04 19:21:15,930 [DEBUG] mongo_connector.oplog_manager:654 - OplogThread: oplog checkpoint updated to Timestamp(1459797674, 1)
When I look at the object in ES though, it's still the same as the original.
:+1: I am having a similar issue.
Same issue happening here. Can we get more eyes on this?
@aherlihy Okay, this issue is definitely related to PR #382
I tried changing a top-level property on my object and that update did indeed propagate.
@nVitius Hi, I just pushed a commit that fixes the issues with nested fields in --include-fields and adds support for --exclude-fields. If you retry with the most recent version of Mongo-Connector (off master), does that fix your issue?
Sorry for the delay in getting back to you!
Hi, I'm going to close this ticket due to inactivity but please feel free to reopen or file a new ticket if you are still having problems. Thanks!
I can't seem to get mongo-connector to replicate my updates/deletes to Elasticsearch.
I'm using mongo-connector 2.3.0 w/ elastic-doc-manager2 and elasticsearch 2.1.1