BUG: 'Index with name: cyg_agg_opt already exists with a different name' on server XX.X.X.XX:XXXXX

[smartcitydevops]

We have detected that there are many errors when persisting data in cygnus_sth, according to these repeated logs:

11/4/20
12:22:34.197 PM 
time=2020-11-04T11:22:34.197Z | lvl=WARN | corr=4251e79d-d2d0-4d6f-803c-03f1b1a1ebfc | trans=a5759776-4ef9-4fd6-81cb-10d469940db6 | srv=servicenamel | subsrv=/ | op=createCollection | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.mongo.MongoBackendImpl[130] : Error in collection sth_/.aggr creating index ex=Command failed with error 85 (IndexOptionsConflict): 'Index with name: cyg_agg_opt already exists with a different name' on server 10.0.0.19:27017. The full response is {"operationTime": {"$timestamp": {"t": 1604488954, "i": 58}}, "ok": 0.0, "errmsg": "Index with name: cyg_agg_opt already exists with a different name", "code": 85, "codeName": "IndexOptionsConflict", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1604488954, "i": 58}}, "signature": {"hash": {"$binary": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type": "00"}, "keyId": {"$numberLong": "0"}}}}
host = machine2  source = /var/log/cygnus/cygnus_sth.log
11/4/20
12:22:34.172 PM 
time=2020-11-04T11:22:34.172Z | lvl=WARN | corr=4251e79d-d2d0-4d6f-803c-03f1b1a1ebfc | trans=fbf2878c-9be1-4d3e-b7db-9f6ee0a9f5b2 | srv=servicename| subsrv=/ | op=createCollection | comp=Cygnus | msg=com.telefonica.iot.cygnus.backends.mongo.MongoBackendImpl[130] : Error in collection sth_/.aggr creating index ex=Command failed with error 85 (IndexOptionsConflict): 'Index with name: cyg_agg_opt already exists with a different name' on server 10.0.0.19:27017. The full response is {"operationTime": {"$timestamp": {"t": 1604488954, "i": 37}}, "ok": 0.0, "errmsg": "Index with name: cyg_agg_opt already exists with a different name", "code": 85, "codeName": "IndexOptionsConflict", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1604488954, "i": 37}}, "signature": {"hash": {"$binary": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type": "00"}, "keyId": {"$numberLong": "0"}}}}
host = machine2 source = /var/log/cygnus/cygnus_sth.log

We have detected this iuise when upgading the mongodb version from 3.6 to 4.2. Just this change.

The existing indexes in both collections are:

cb_rs0:PRIMARY> db['sth_/'].getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "sth_servicename.sth_/"
        },
        {
                "v" : 2,
                "key" : {
                        "entityId" : 1,
                        "entityType" : 1,
                        "attrName" : 1,
                        "recvTime" : 1
                },
                "name" : "entityId_1_entityType_1_attrName_1_recvTime_1",
                "ns" : "sth_servicename.sth_/"
        }
]
cb_rs0:PRIMARY> db['sth_/.attr'].getIndexes()
[ ]
cb_rs0:PRIMARY> db['sth_/.aggr'].getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "sth_servicename.sth_/.aggr"
        },
        {
                "v" : 2,
                "key" : {
                        "_id.entityId" : 1,
                        "_id.entityType" : 1,
                        "_id.attrName" : 1,
                        "_id.resolution" : 1,
                        "_id.origin" : 1
                },
                "name" : "_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1",
                "ns" : "sth_servicename.sth_/.aggr"
        },
        {
                "v" : 2,
                "key" : {
                        "_id.origin" : 1
                },
                "name" : "_id.origin_1",
                "ns" : "sth_servicename.sth_/.aggr",
                "background" : true,
                "expireAfterSeconds" : 60
        }
]

We think that cyg_agg_opt is an expected index by cygnus_sth . Please update in order to perform any action

[fgalan]

The problems is probably due to Cygnuss trying to create an index which one of the following key patterns:

{
                        "_id.entityId" : 1,
                        "_id.entityType" : 1,
                        "_id.attrName" : 1,
                        "_id.resolution" : 1,
                        "_id.origin" : 1
                }

or

{
                        "_id.origin" : 1
                }

Cygnus tryes to use the name cyg_agg_opt for that index, but the problem doesn't seem actually related with index names. I may say that it is related with they key pattern Cygnus tries to use.

[AlvaroVega]

The reported log is just a warn....is not a bug. It's an undesirable but expected situation.

[fgalan]

Just in case it can be useful, some code hint found by @mrutid yesterday:

It seems that https://github.com/telefonicaid/fiware-cygnus/commit/713913ecc2a4b24489fa86d0c8297c669a46e03a adds the catch that is printing the WARN trace:

        } catch (Exception e) {
            LOGGER.warn("Error in collection " + collectionName + " creating index ex=" + e.getMessage());
        }

That commit belongs to PR https://github.com/telefonicaid/fiware-cygnus/issues/1975 and, according to git, has been around since Cygnus 1.16.0 (september 2019)