Open TLINDEN opened 1 year ago
Ok, I tried the same process as above but using mongodb 5.0. That failed as well. First it had the same problem with the non-existing configmap. So I added it now to my deployment.
However, now the pre-update pod fails to execute:
% k logs rocketchat-fitstest-rocketchat-pre-upgrade
Current Mongosh Log ID: 64117dd3c10c36597a8d184c
Connecting to: mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB: 4.4.11
Using Mongosh: 1.8.0
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.
Loading file: /tmp/verifyMongodb.js
MongoServerError: not authorized on admin to execute command { getParameter: 1, featureCompatibilityVersion: 1, lsid: { id: UUID("0d4d5821-8149-4857-89e5-312ea8c2a2b0") }, $clusterTime: { clusterTime: Timestamp(1678867922, 1), signature: { hash: BinData(0, A18E69A731CE183BF26858E13F89D25DBA829ABD), keyId: 7210672631631052805 } }, $db: "admin" }
The reason is, that the user rocketchat
is no admin user, there's only root:
rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers()
[
{
"_id" : "admin.root",
"userId" : UUID("dcd90659-2364-4ed9-b446-15a511a96511"),
"user" : "root",
"db" : "admin",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
],
"mechanisms" : [
"SCRAM-SHA-1",
"SCRAM-SHA-256"
]
}
]
So, how is this supposed to work?
Next iteration:
I've setup a test pod using the same image as the pre-update pod:
---
apiVersion: v1
kind: Pod
metadata:
annotations:
name: test
spec:
containers:
- command:
- sleep
- "1010101010101010"
image: docker.io/bitnami/mongodb:5.0.15-debian-11-r6
imagePullPolicy: IfNotPresent
name: mongosh
logged into it, created the /tmp/verifyMongodb.js
file, configured the proper connection string and tried to execute what the pre-update pod does:
I have no name!@test:/$ export MONGODB_HOST="mongodb://root:******@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6
Current Mongosh Log ID: 64119862238d764edda38ebd
Connecting to: mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
MongoServerError: Authentication failed.
However, on the mongodb pod I actually CAN use the root
user:
kubectl exec -it rocketchat-fitstest-mongodb-0 -- mongo -u root -p ****** --host localhost
Defaulted container "mongodb" out of: mongodb, volume-permissions (init)
MongoDB shell version v4.4.11
connecting to: mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("1ce823d0-226f-41ba-b442-7a38c9c5da0a") }
MongoDB server version: 4.4.11
---
The server generated these startup warnings when booting:
2023-03-15T07:48:47.794+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-03-15T07:48:48.732+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
2023-03-15T07:48:48.732+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers()
[
{
"_id" : "admin.root",
"userId" : UUID("dcd90659-2364-4ed9-b446-15a511a96511"),
"user" : "root",
"db" : "admin",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
],
"mechanisms" : [
"SCRAM-SHA-1",
"SCRAM-SHA-256"
]
}
]
But even IF that would work, the helm chart doesn't use the root
user, but the rocketchat
user, which doesn't even exist. Since I found no way to modify the chart to use root
I tried just adding the rocketchat user as admin using the password defined in the deployment.
{
"_id" : "admin.rocketchat",
"userId" : UUID("546c47df-9043-48de-bad0-3f3e8f08d7af"),
"user" : "rocketchat",
"db" : "admin",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
],
"mechanisms" : [
"SCRAM-SHA-1",
"SCRAM-SHA-256"
]
},
But even THEN it doesn't work:
I have no name!@test:/$ export MONGODB_HOST="mongodb://rocketchat:******@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6
Current Mongosh Log ID: 64119a54ef9d794cfe1dfb75
Connecting to: mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB: 4.4.11
Using Mongosh: 1.8.0
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.
Loading file: /tmp/verifyMongodb.js
MongoServerError: not authorized on admin to execute command { getParameter: 1, featureCompatibilityVersion: 1, lsid: { id: UUID("cc669895-7271-4f39-bf1f-ba2ff67aa1cf") }, $clusterTime: { clusterTime: Timestamp(1678875213, 1), signature: { hash: BinData(0, 3D14859A65F46FF974E9BBE6649C8FEA30315A7D), keyId: 7210672631631052805 } }, $db: "admin" }
So, please help me, the whole thing looks hopeless to me!
Thanks in advance, Tom
PS: I found a way to get the verify Script executed:
I have no name!@test:/$ export MONGODB_HOST="mongodb://root:******@rocketchat-fitstest-mongodb-headless:27017/admin?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6; echo $?
Current Mongosh Log ID: 64119c81411695734767a2d1
Connecting to: mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/admin?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB: 4.4.11
Using Mongosh: 1.8.0
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
------
The server generated these startup warnings when booting
2023-03-15T07:48:47.794+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-03-15T07:48:48.732+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
2023-03-15T07:48:48.732+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
------
------
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------
Loading file: /tmp/verifyMongodb.js
0
The important thing here is that I am not using the rocketchat
database but admin
. This even works with the rocketchat user (if you give it admin permissions, see above).
It continues:
Now that I know that in principle upgrading mongodb from 4.4 to 5.0 seems to be supported and since I deployed the configmap myself anyway, I replaced the JS code with a bare quit(0)
. That way now the upgrade runs through, at least for mongodb.
However, now I'm getting this error from the rocketchat pod:
Some indexes for collection 'rocketchat_credential_tokens' could not be created:
An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: true, expireAfterSeconds: 0 }, existing index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: 1, expireAfterSeconds: 0 }
LocalStore: store created at
LocalStore: store created at
LocalStore: store created at
Error creating index: livechat_priority -> { name: 1 } {
unique: true,
partialFilterExpression: { '$and': [ [Object], [Object] ] }
} MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", partialFilterExpression: { $and: [ { name: { $exists: true } }, { name: { $gt: "" } } ] } }, existing index: { v: 2, unique: true, key: { name: 1 }, name: "name_1" }
at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:230:30)
at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
at MessageStream.emit (events.js:400:28)
at MessageStream.emit (domain.js:475:12)
at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
at writeOrBuffer (internal/streams/writable.js:358:12)
at MessageStream.Writable.write (internal/streams/writable.js:303:10)
at Socket.ondata (internal/streams/readable.js:731:22)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:475:12)
at addChunk (internal/streams/readable.js:293:12)
at readableAddChunk (internal/streams/readable.js:267:9)
at Socket.Readable.push (internal/streams/readable.js:206:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
ok: 0,
code: 85,
codeName: 'IndexOptionsConflict',
'$clusterTime': {
clusterTime: new Timestamp({ t: 1678876192, i: 3 }),
signature: {
hash: new Binary(Buffer.from("8ef9a22c7f50037d8e65fdab1d0b947fbe11c3ce", "hex"), 0),
keyId: new Long("7210672631631052805")
}
},
operationTime: new Timestamp({ t: 1678876192, i: 3 }),
[Symbol(errorLabels)]: Set(0) {}
}
strict mode: use allowUnionTypes to allow union type keyword at "#/properties/value" (strictTypes)
However - rocketchat comes up anyway without further errors and seems to work.
Howdy,
my deployment is unfortunately a little outdated, so I thaught it might be a good idea to upgrade rocketchat first, keep mongodb version and upgrade that later.
So I specified helm chart version
5.1.0
but mongodb image tag4.4.11-debian-10-r12
and deployed.The chart then tried to deploy an upgrade pod, which failed to start:
The problem was, that said configmap first appeared in 5.4.3, so it's not installed here, but the pre-upgrade pod just expects it to be there.
My next try was to just manually deploy that script (this would be required only once anyway). Now the pre-upgrade pod came up, but failed with:
because
mongosh
seems not to be installed in the 4.4 image.My next try will be to directly upgrade to mongodb 5.0 and see what happens. But I think this behavior should be fixed anyway somehow.