telefonicaid / fiware-orion

Context Broker and CEF building block for context data management, providing NGSI interfaces.
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md
GNU Affero General Public License v3.0
210 stars 265 forks source link

Possible Bug: improper Index creation in MongoDB #3484

Closed jicarretero closed 5 years ago

jicarretero commented 5 years ago

I've created 3000 entities using IoT Agent Json (Let's say I have 3000 cars). So, In mongoDB we have this representation:

{ "_id" : { "id" : "car0001", "type" : "thing", "servicePath" : "/car_measurement" }, "attrNames":...

This means the _id is compound of 4 fields: id, type, type, servicePath.

I can see in MongoDB that there are some indexes created on _id:

> db.entities.getIndexes();
[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "orion-car.entities"
    },
....

However, the query made by contextBroker is this one: { _id.id: "car2623", _id.type: "thing", _id.servicePath: { $in: [ /^/car_measurement$/ ] } } Which makes the index "id" useless. In fact there is no search on _id but on _id.id, _id.type, ... --- So, my suggestion would be to create an index on "_id.id".

This improves quite a lot Orion ContexBroker response time.

fgalan commented 5 years ago

According to https://fiware-orion.readthedocs.io/en/master/admin/perf_tuning/index.html#database-indexes

Orion Context Broker doesn't create any index in any database collection (with two exceptions, described at the end of this section), to give flexibility to database administrators. Take into account that index usage involves a tradeoff between read efficiency (usage of indexes generally speeds up reads) and write efficiency (the usage of indexes slows down writes) and storage (indexes consume space in database and mapped RAM memory) and it is the administrator (not Orion) who has to decide what to prioritize. However, in order to help administrators in this task, the following indexes are recommended:

  • Collection entities _id.id _id.type _id.servicePath attrNames creDate

Note that the indexes you cite are included in this recommendation.

jicarretero commented 5 years ago

In fact, 3 indexes are created instead of the 2 described in the documentation. The 3rd one is:

{
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "orion.entities"
    },

And this one doesn't seem to be of use.

Thank you.

fgalan commented 5 years ago

That index is not related with Orion, but with MongoDB. MongoDB always creates an index for _id in every collection. As far as a I know, there is no way of avoid its creation or removing it.

fgalan commented 5 years ago

Once clarification has been done I think this issue can be closed. However, if I'm wrong please @jicarretero tell me so and we will reopen it.