Cannot Restore Rocket Chat 6.11 from EBS Snapshot

ThatTallProgrammer commented 1 month ago

Description:

I recently upgraded to Rocket Chat 6.11. This was a fairly massive upgrade from version 3 to version 6 (I know). I used the instructions at https://docs.rocket.chat/docs/updating-rocketchat. This involved three instances of having to drop and recreate indexes, but from reading the Rocket Chat issues, this seems to be expected.

The upgrade works fine and the server will run so long as the database doesn't need to be restored. I use AWS EBS for MongoDB as part of an EKS deployment. If the database needs to be restored, we use an EBS snapshot. This seems to break many of indexes producing the following in the logs.

See Broken Index Log Sample below.

I handled this the way I've been handling broken indexes the entire time. I dropped them, restarted the Rocket Chat pod, and let Rocket Chat rebuild them. This seemed to work, however, the server now just doesn't start. I've waiting for well over two hours, but the logs never go past this point.

See Server Fails to Start Log Sample below.

I have tried to restore the database completely from a mongodump and I get the same result. Are there any known incompatibilities with EBS snapshots? This only happens when I try to restore from a snapshot, which is the only common thread. Anyone have any idea what is going on?

Steps to reproduce:

Create an EKS cluster
Configure a Rocket Chat deployment
Configure a MongoDB deployment using EBS for the database volume
Take a snapshot of the EBS volume
Attempt to restore MongoDB from an EBS snapshot
Restart Rocket Chat

Expected behavior:

Server should start as expected and show the server startup confirmation in the logs.

Actual behavior:

Rocket Chat appears to bootstrap just fine, but the server never seems to start.

Server Setup Information:

Version of Rocket.Chat Server: 6.11.0
License Type: CE
Operating System: RHEL 9
Deployment Method: Kubernetes (EKS)
Number of Running Instances: 1
DB Replicaset Oplog: Unsure how to obtain this
NodeJS Version: Default for version 6.11.0
MongoDB Version: 5.0.27

Client Setup Information

Desktop App or Browser Version: N/A
Operating System: N/A

Additional context

Relevant logs:

Broke Index Log Sample

Some indexes for collection 'rocketchat_analytics' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { room._id: 1, date: 1 }, name: "room._id_1_date_1", partialFilterExpression: { type: "rooms" } }, existing index: { v: 2, unique: true, key: { room._id: 1, date: 1 }, name: "room._id_1_date_1", ns: "rocketchat.rocketchat_analytics" }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { room._id: 1, date: 1 }, name: "room._id_1_date_1", partialFilterExpression: { type: "rooms" } }, existing index: { v: 2, unique: true, key: { room._id: 1, date: 1 }, name: "room._id_1_date_1", ns: "rocketchat.rocketchat_analytics" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Some indexes for collection 'rocketchat_credential_tokens' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: true, expireAfterSeconds: 0 }, existing index: { v: 1, key: { expireAt: 1 }, name: "expireAt_1", ns: "rocketchat.rocketchat_credential_tokens", sparse: 1, expireAfterSeconds: 0 }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: true, expireAfterSeconds: 0 }, existing index: { v: 1, key: { expireAt: 1 }, name: "expireAt_1", ns: "rocketchat.rocketchat_credential_tokens", sparse: 1, expireAfterSeconds: 0 }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 4)
Some indexes for collection 'rocketchat_integration_history' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1", expireAfterSeconds: 2592000 }, existing index: { v: 1, key: { _updatedAt: 1 }, name: "_updatedAt_1", ns: "rocketchat.rocketchat_integration_history" }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1", expireAfterSeconds: 2592000 }, existing index: { v: 1, key: { _updatedAt: 1 }, name: "_updatedAt_1", ns: "rocketchat.rocketchat_integration_history" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 6)
Some indexes for collection 'rocketchat_nps_vote' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1" }, existing index: { v: 2, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1", ns: "rocketchat.rocketchat_nps_vote" }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1" }, existing index: { v: 2, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1", ns: "rocketchat.rocketchat_nps_vote" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 8)
Some indexes for collection 'rocketchat_message' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { editedAt: 1 }, name: "editedAt_1", sparse: true }, existing index: { v: 1, key: { editedAt: 1 }, name: "editedAt_1", ns: "rocketchat.rocketchat_message", sparse: 1 }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { editedAt: 1 }, name: "editedAt_1", sparse: true }, existing index: { v: 1, key: { editedAt: 1 }, name: "editedAt_1", ns: "rocketchat.rocketchat_message", sparse: 1 }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 10)
Some indexes for collection 'rocketchat_room' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", sparse: true }, existing index: { v: 1, unique: true, key: { name: 1 }, name: "name_1", ns: "rocketchat.rocketchat_room", sparse: 1 }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", sparse: true }, existing index: { v: 1, unique: true, key: { name: 1 }, name: "name_1", ns: "rocketchat.rocketchat_room", sparse: 1 }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 12)
Some indexes for collection 'rocketchat_subscription' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { desktopNotifications: 1 }, name: "desktopNotifications_1", sparse: true }, existing index: { v: 1, key: { desktopNotifications: 1 }, name: "desktopNotifications_1", ns: "rocketchat.rocketchat_subscription", sparse: 1 }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { desktopNotifications: 1 }, name: "desktopNotifications_1", sparse: true }, existing index: { v: 1, key: { desktopNotifications: 1 }, name: "desktopNotifications_1", ns: "rocketchat.rocketchat_subscription", sparse: 1 }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 14)
Some indexes for collection 'rocketchat_uploads' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { rid: 1 }, name: "rid_1", sparse: true }, existing index: { v: 1, key: { rid: 1 }, name: "rid_1", ns: "rocketchat.rocketchat_uploads" }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { rid: 1 }, name: "rid_1", sparse: true }, existing index: { v: 1, key: { rid: 1 }, name: "rid_1", ns: "rocketchat.rocketchat_uploads" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 16)
Some indexes for collection 'rocketchat_room' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { open: 1 }, name: "open_1", sparse: true }, existing index: { v: 1, key: { open: 1 }, name: "open_1", sparse: 1, ns: "rocketchat.rocketchat_room" }
(node:1) UnhandledPromiseRejectionWarning: MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { open: 1 }, name: "open_1", sparse: true }, existing index: { v: 1, key: { open: 1 }, name: "open_1", sparse: 1, ns: "rocketchat.rocketchat_room" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 18)
(node:1) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
Some indexes for collection 'rocketchat_livechat_priority' could not be created:
    An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", partialFilterExpression: { $and: [ { name: { $exists: true } }, { name: { $gt: "" } } ] } }, existing index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", ns: "rocketchat.rocketchat_livechat_priority" }
=== UnHandledPromiseRejection ===
MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", partialFilterExpression: { $and: [ { name: { $exists: true } }, { name: { $gt: "" } } ] } }, existing index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", ns: "rocketchat.rocketchat_livechat_priority" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:231:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at TLSSocket.ondata (internal/streams/readable.js:731:22)
    at TLSSocket.emit (events.js:400:28)
    at TLSSocket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:206:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
    at TLSWrap.callbackTrampoline (internal/async_hooks.js:130:17) {
  ok: 0,
  code: 85,
  codeName: 'IndexOptionsConflict',
  '$clusterTime': {
    clusterTime: new Timestamp({ t: 1723674853, i: 1 }),
    signature: {
      hash: new Binary(Buffer.from("923831f88004b405b0ef45cbd765f4f10f4a94d3", "hex"), 0),
      keyId: new Long("7362398007834181636")
    }
  },
  operationTime: new Timestamp({ t: 1723674853, i: 1 }),
  [Symbol(errorLabels)]: Set(0) {}
}
---------------------------------
Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process

Server Never Starts Log Sample

(node:1) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
{"level":40,"time":"2024-08-14T22:24:23.479Z","pid":1,"hostname":"rocketchat-544fcb5dc4-gjn95","name":"VoIPAsteriskService","msg":"Voip is not enabled. Cant start the service"}
(node:1) [MONGODB DRIVER] Warning: cursor.count is deprecated and will be removed in the next major version, please use `collection.estimatedDocumentCount` or `collection.countDocuments` instead 
{"level":30,"time":"2024-08-14T22:24:25.488Z","pid":1,"hostname":"rocketchat-544fcb5dc4-gjn95","name":"CAS","msg":"Enabling CAS login service"}

reetp commented 1 month ago

Hi, and thanks for reporting.

Just to tie this together as there's bits everywhere which makes it hard to track.

I note you foro post here:

https://forums.rocket.chat/t/rocket-chat-does-not-start-after-6-3-5-update-from-6-3-0/17810/5?u=reetp

Related to this bug here:

https://github.com/RocketChat/Rocket.Chat/issues/30398

However I don't think this is related.

Your upgrade appeared to be succesful.

Your issue is this which is unrelated to the above:

The upgrade works fine and the server will run so long as the database doesn't need to be restored. I use AWS EBS for MongoDB as part of an EKS deployment. If the database needs to be restored, we use an EBS snapshot.

So a read about these reveals this:

https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html

Importantly I think is this:

The replicated volume loads data in the background so that you can begin using it immediately. If you access data that hasn't been loaded yet, the volume immediately downloads the requested data from Amazon S3, and then continues loading the rest of the volume's data in the background.

I'm not sure this method is considered "supported".

Have you tested with a stock mongodump/restore?

We'd like to isolate EBS as the culprit here.

ThatTallProgrammer commented 1 month ago

Hi, reetp.

Yes, I have tested using a mongodump/restore both immediately after upgrade when the app is working and from a MongoDB instance that was restored from an EBS snapshot. My method for doing this is to drop the rocketchat database on the existing MongoDB and then do a mongorestore. I make sure that no rocket chat instances are connected during the restore process.

Restoring the MongoDB instance from the dump that was created when the app is running without error seems to work. This is true even if the MongoDB instance has been restored from an EBS snapshot.

Regarding the issue I foro posted, while the exact point in which the issue arises seems different, the behavior is identical. It's the only issue I could find that showed the behavior I'm experiencing. There are several similarities including a container deployment, albeit with different runtimes, and MongoDB versions. Could it be possible that it's the same issue manifesting in two different ways?

reetp commented 4 weeks ago

Sorry I have just read your post a couple of times and I can't quite get my head around it :fearful:

Earlier you said:

I have tried to restore the database completely from a mongodump and I get the same result. Are there any known incompatibilities with EBS snapshots? This only happens when I try to restore from a snapshot, which is the only common thread. Anyone have any idea what is going on?

But then you have just said:

Restoring the MongoDB instance from the dump that was created when the app is running without error seems to work. This is true even if the MongoDB instance has been restored from an EBS snapshot.

Those statements seem to contradict one another!

Can you please clarify the results of the following backups on the database only:

Mongo dump/mongo restore - the accepted method. No EBS involved.

I suspect the mongo dump/restore is fine (apart from any issues you may have experienced trying to downgrade it as mentioned above)

EBS backup/restore only

I also suspect the EBS backup/restore may be problematic. The only solution I might see there is a full restore of the DB without Rocket running until you know the DB is complete.

Alternatively you take a backup of both Rocket AND the db at the same time for restore. No idea whether this would work.

I can't see any AWS docs on EBS/Mongo to provide any advice either. Let us know if you see anything relevant.

ThatTallProgrammer commented 3 weeks ago

Those statements seem to contradict one another!

To clarify, what works for me is making a dump from mongo while the instance is healthy and operational. I can restore to that point in time. What I was trying to say is that if I restore from an EBS snapshot, that action still produces the index errors and eventual server failing to start issue I described. However, on the broken MongoDB, I am able to drop the rocketchat database, do a mongorestore, and bring the instance back to operational.

Can you please clarify the results of the following backups on the database only

Mongodump/restore works so long as the dump was taken before restoring from an EBS snapshot.

I suspect the mongo dump/restore is fine (apart from any issues you may have experienced trying to downgrade it as mentioned above)

I may be misinterpreting this, but I have not tried to downgrade the Rocket Chat instance at any point using mongorestore. I have reverted to version 3 using an EBS snapshot which seemed to work fine.

I also suspect the EBS backup/restore may be problematic. The only solution I might see there is a full restore of the DB without Rocket running until you know the DB is complete.

I did a fairly thorough test on this yesterday. I restored MongoDB using an EBS snapshot without anything connected to it. Since you suspected that lazy loading may be an issue, I dumped the entire volume into /dev/null so that every part of the volume was read at least once, and therefore nothing should have been lazily loaded. Unfortunately, this did not resolve the issue.

reetp commented 3 weeks ago

So it would appear that it is all fine using mongodump/restore but, the moment EBS is involved you have issues.

Is that correct?

More on supported mongo backup options here as it would seem to be mongo backup/restore issue rather than a Rocket.Chat issue

https://www.mongodb.com/docs/manual/core/backups/

I will try and ask the dev ream if they have any comments.

github-actions[bot] commented 1 week ago

This issue has been marked as stale because there has been no further activity in the last 10 days. If the issue remains stale for the next 4 days (a total of 14 days with no activity), then it will be assumed that the question has been resolved and the issue will be automatically closed.

RocketChat / Rocket.Chat