how to denote datatype "nested" instead of Object - what differentiate object and nested in elastic search mapping in CrawlerData.

[raj92]

I am stuck while defining the mapping for a field (files) type that is nested rather then Object. I am looking into solution to define the files type = "nested" otherwise filter will not work.

As we are to define the mapping in json file and preprocessor pipeline - we are already creating Array Object [{}] but how to denote "nested" - what differentiate object and nested in elastic search mapping in CrawlerData.

They must be some way in "node-mongodb-es-connector" plugin which help us do so.

because [{}] is purely array object.

Please help.

[raj92]

Any update on this ? Please reply.

[zhr85210078]

I think you should use 'Foreach Processor' and 'JSON Processor' to create a pipeline. Like this:

PUT _ingest/pipeline/yourpip { "description" : "test", "processors" : [ { "foreach": { "field": "yourArrayField", "processor": { "json": { "field": "yourObjField" } } } } ] }

I hope I can help you.

[raj92]

As per the suggestion the PIPELINE BODY will look like this ::

PUT :: localhost:9200/_ingest/pipeline/convert-object-nested-pipeline { "description" : "files to be array objects", "processors" : [ { "foreach": { "field": "files", "processor": { "json": { "field": "files" } } } } ] }

Response ::

{ "acknowledged": true }

My CrawlerData/userfiles.json file ::

{ "mongodb": { "m_database": "myTest", "m_collectionname": "userfiles", "m_filterfilds": { "version": "2.0" }, "m_returnfilds": { "files" : [{"fileId" : "8eecdf24-a783-4430-abfc-7bb2c6487cfb"}], "orgId" : "text", "directory" : "type", "size" : "integer", "userName" : "text", "groupId": "type" }, "m_extendfilds": { "bA": "this is a extend fild bA", "bB": "this is a extend fild bB" }, "m_extendinit": { "m_comparefild": "_id", "m_comparefildType": "ObjectId", "m_startFrom": "2018-09-17 13:44:00", "m_endTo": "2018-09-17 16:27:51" }, "m_connection": { "m_servers": [ "localhost:27017" ], "m_authentication": { "username": "test", "password": "test123", "authsource": "myTest", "replicaset": "test",
"ssl": false } }, "m_url": "mongodb://test:test123@localhost:27017/myTest?authSource=myTest", "m_documentsinbatch": 5000, "m_delaytime": 1000, "max_attachment_size":5242880 }, "elasticsearch": { "e_index": "userfiles", "e_type": "userfiles", "e_connection": { "e_server": "http://localhost:9200", "e_httpauth": { "username": "EsAdmin", "password": "pass1234" } }, "e_pipeline": false, "e_iscontainattachment": false } }

Please note in the above json file if i remove pipeline name/ID it stoping working (not creating index into Elastic Search). But it work when i make it false.

Our purpose it make the file array object filterable. Now the object in the file array are pure object - we have to make it nested.

Therefore, we are trying to define the DATATYPE of files into NESTED - To achieve full text search (filter 100%).

So that we can search the nested document after adding mapping like this :

GET workspaces_userfiles/_search?pretty { "query": { "nested": { "path": "files", "query": { "match": { "files.name": "a" } }, "inner_hits": {}

But as per your suggestion, we are again making it JSON object.

May be the mapping structure and our json data structure help you in understanding.

{ "mappings": { "properties": { "files": { "type": "nested", "properties": { "shared": {"type": "text"}, "s3url": {"type": "text"}, "lastDownloadedOn": {"type": "text"}, "type": {"type": "text"}, "createdOn": {"type": "text"}, "path": {"type": "text"}, "lastUpdated": {"type": "text"}, "readablePdfPath": {"type": "text"}, "size": {"type": "integer"}, "encrypted": {"type": "text"}, "name": {"type": "text"}, "thumbPath": {"type": "text"}, "fileId": {"type": "text"} } }, "orgId" : {"type": "text"}, "directory" : {"type": "text"}, "size" : {"type": "integer"}, "userName" : {"type": "text"}, "groupId": {"type": "text"}, "timestamp" : {"type": "text"} } } }

{ "_id" : ObjectId("5d6e3e23dc41902ae4152c2a"), "orgId" : "o20190828092008798", "directory" : "pratirumala", "size" : -201920, "files" : [ { "fileId" : "8eecdf24-a783-4430-abfc-7bb2c6487cfb", "name" : "a1", "type" : "Folder", "size" : 0, "path" : "pratirumala/a1", "shared" : "No", "createdOn" : "2019-10-26T05:03:25.778Z", "lastUpdated" : "2019-10-26T05:03:25.778Z", "encrypted" : "No" }, { "fileId" : "525631de-95e5-43a1-8756-e356ce4cf4b4", "name" : "f01", "type" : "Folder", "size" : 0, "path" : "pratirumala/f01", "shared" : "No", "createdOn" : "2019-11-07T04:13:15.476Z", "lastUpdated" : "2019-11-07T04:13:15.476Z", "encrypted" : "No" } ] }

Hope you understand what is our issue.

We have also tried to convert the field type : https://www.elastic.co/guide/en/elasticsearch/reference/current/convert-processor.html?utm_source=hacpai.com

But "nested" data type not supported.

Please help.

Also, Please revert if any question.