K8SPSMDB-1058: A minor missing privileges issue caused flooding MongoDB logs with “Checking authorization failed” errors

[tm-nadavsh]

Report

I had issue with flooding logs of “Checking authorization failed”, saw on new version release that operator version 1.16 should fix it https://perconadev.atlassian.net/browse/K8SPSMDB-1058 When upgrading to psmdb-operator 1.16.0 , the issue is still happening.

Also happening on a fresh installation on version 1.16.0 so looks like the bug is not fixed

More about the problem

I understood that the issue is the clusterMonitor user has the explain role that is missing a privilege of collection system.version, when trying to update the role via:

db.updateRole("explainRole", {
  privileges: [
    {
      resource: {
        db: "",
        collection: ""
      },
      actions: [
        "collStats",
        "dbHash",
        "dbStats",
        "find",
        "listIndexes",
        "listCollections"
      ]
    },
    {
      resource: {
        db: "",
        collection: "system.profile"
      },
      actions: [
        "dbStats",
        "collStats",
        "indexStats"
      ]
    },
    {
      resource: {
        db: "",
        collection: "system.version"
      },
      actions: [
        "find"
      ]
    }
  ],
  roles: []
})

It changes and then immediately returns to its old values due to psmdb-operator, also tried stopping psmdb-operator, which makes the changes last, then when turning it on it reverts it again... Also tried dropping the role and recreating it, creating a new role (which was succesful) and then adding it to the cluster monitor, all changes were reverted by psmdb-operator in a matter of seconds.

Also, tried to re deploy psmdb as a whole, deleted operator, percona server and pmm, deleted all pvc's and installed fresh with operator on version 1.16.0, still got issue.

Steps to reproduce

1. install psmdb (version 1.14.3)
2. install pmm (version 1.2.3)
3. install psmdb-operator (version 1.15.0)
4. upgrade psmdb-operator to version 1.16.0

Also:

1. install psmdb (version 1.14.3)
2. install pmm (version 1.2.3)
3. install psmdb-operator (version 1.16.0)

Versions

1. Kubernetes 1.28 on EKS
2. Operator 1.16.0
3. Database 1.14.3

Anything else?

No response

[hors]

Hi @tm-nadavsh, could you please provide explainRole that you have in your PSMDB cluster. And please provide full error massage. Thank you.

[tm-nadavsh]

Sure @hors, explainRole:

{
    "_id" : "admin.explainRole",
    "role" : "explainRole",
    "db" : "admin",
    "privileges" : [
        {
            "resource" : {
                "db" : "",
                "collection" : ""
            },
            "actions" : [
                "collStats",
                "dbHash",
                "dbStats",
                "find",
                "listCollections",
                "listIndexes"
            ]
        },
        {
            "resource" : {
                "db" : "",
                "collection" : "system.profile"
            },
            "actions" : [
                "collStats",
                "dbStats",
                "indexStats"
            ]
        }
    ],
    "roles" : [

    ]
}

looks like its missing the system.version privilege Error message:

{"t":{"$date":"2024-09-22T08:41:48.073+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn12","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { find: \"system.version\", filter: { _id: \"shardIdentity\" }, limit: 1, singleBatch: true, lsid: { id: UUID(\"9757fa79-9a8e-4eab-98d3-0659e2467c51\") }, $clusterTime: { clusterTime: Timestamp(1726994507, 1), signature: { hash: BinData(0, 4028119521C2174B8A52BE8BBF7C286C9CCEF2A0), keyId: 7376354637206519816 } }, $db: \"admin\", $readPreference: { mode: \"primaryPreferred\" } }"}}}

When listing sessions i see that this UID is of the clusterMonitor user:

{
    "_id" : {
        "id" : UUID("9757fa79-9a8e-4eab-98d3-0659e2467c51"),
        "uid" : BinData(0, "DMLJ6czxPet7fkqSAZxB3DmnBLY4s0sG.. 8 more bytes")
    },
    "lastUse" : ISODate("2024-09-22T08:42:05.536+0000"),
    "user" : {
        "name" : "clusterMonitor@admin"
    }
}

Again, when trying to update anything related to the role or the user it reverts my changes automatically, tried updating to this same result.

[hors]

@tm-nadavsh Did you try to use crVersion >= 1.16.0? I can't reproduce this issue. I have the following role:

    {
      _id: 'admin.explainRole',
      role: 'explainRole',
      db: 'admin',
      privileges: [
        {
          resource: { db: '', collection: '' },
          actions: [
            'collStats',
            'dbHash',
            'dbStats',
            'find',
            'listCollections',
            'listIndexes'
          ]
        },
        {
          resource: { db: '', collection: 'system.profile' },
          actions: [ 'collStats', 'dbStats', 'indexStats' ]
        },
        {
          resource: { db: 'admin', collection: 'system.version' },
          actions: [ 'find' ]
        }

Additional role was added only for crVersion >= 1.16.0.

[tm-nadavsh]

@hors Thank you!, that was the missing piece i had my crVersion hardcoded with the value of 1.15.0, updated to 1.16.0 and it works now. Maybe would be nice to add this to the article in here about the new version or some sort of a migration process.

[tm-nadavsh]

Also worth replaying here https://forums.percona.com/t/cannot-auto-discover-databases-and-collections-cannot-list-the-collections-checking-authorization-failed/24172/4 as he also complained and this looks the same issue