Deleting from mongo

[andymcollins]

HI This is great App.

I have it working for inserting data in mongo and it syncs to Elasticsearch, however in MongoData there is a _id field which it didnt like so i

addeddocument.MongoID = document._id;
document = _.omit(document, '_id');

to the transformFunction, which was ok, but when deleting records from Mongo, it gives error

/Users/collia01/Node/rabbittomongo/node_modules/node-elasticsearch-sync/lib/util.js:143 if (!response.found) { ^

TypeError: Cannot read property 'found' of undefined at /Users/collia01/Node/rabbittomongo/node_modules/node-elasticsearch-sync/lib/util.js:143:20 at Object.utils.applyArgs (/Users/collia01/Node/rabbittomongo/node_modules/elasticsearch/src/lib/utils.js:252:17) at wrapper (/Users/collia01/Node/rabbittomongo/node_modules/elasticsearch/node_modules/lodash/index.js:3533:19) at nextTickCallbackWith0Args (node.js:433:9) at process._tickCallback (node.js:362:13)

Is this because the _id has been renamed in elasticseach? Is there a way around this.

Thank you Andy

[toystars]

It's a good thing to remove rename the _id field before indexing in elasticsearch, so that is not the issue here. It's probably from the checks being done to ensure that documents really exist in elasticsearch cluster. Will try to replicate that and see what fix can be applied.

[toystars]

@andymcollins this #5 might be a possible fix. Update, test and let me know if it works as it works down here on my machine without any error.

[andymcollins]

When i remove the _Id from the document using

document = _.omit(document, '_id'); i get message ESMongoSync: Inserted - Index: test Type: external Id: undefined

then when i try to delete a record in MongoDB i get message ESMongoSync: Not Found - Document with id 5728aefd9191198f6ca9991d not found.

if i try to rename the _id field to id with document.id = document._id; i dont get any message to say it has been inserted, and nothing appears in Elasticsearch.

if i convert document_id to string

document.id = document._id.toString(); i get ESMongoSync: Inserted - Index: test Type: external Id: 5728cccbce9cadc97d6ef9b6

but same error when i try to delete ESMongoSync: Not Found - Document with id 5728cccbce9cadc97d6ef9b6 not found.

How do i get the data into Elasticsearch so that any updates/deletes in Mongo are replicated to elasticseach. It seems to just be the ID field. Thanks

[andymcollins]

this is data i'm adding, id is the renamed string coming from MongoDB

{"processedData":"2016-05-03T14:04:18.948Z", "queue":"AndyJobs", "messageID":"1200319843", "oracleProcessedDate":null, "Details":{"VAC_REF":"1200319843", "JOB_TITLE":"string TEXT HERE", "LOCATION":"Surrey", "AGENCY_ID":18462, "EMP_MKT_CD":"EN", "NeutralForm":"String textr", "Discipline":"Construction", "SubDiscipline":"Water", "RemainingSource":"string"}, "id":"5728aff39191198f6ca9997f"}

[andymcollins]

Fixed with my latest pull request