search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.4k stars 10.51k forks source link

Hotchillee Slack Import overwrite issue #20689

Closed tom-gouldson closed 3 years ago

tom-gouldson commented 3 years ago

Context Hotchillee is looking to move its Digital cycling club communications from Slack to a Rocketchat based app. In doing so it hopes to maintain the user experience as much as possible so as not to confuse its user base while using Rocketchats white labelling capability to give it Hotchillee brand and feel. As part of this Hotchillee looks to retain all the historic chat from Slack to make it as seamless as possible for users transferring to the new platform (similar to when Facebook does a major UX overhaul - all the data remains but the look and feel is different and new features are added but it’s still familiar enough not to alienate users).

Description of bug: Before December 15th, when importing data from Slack into Rocketchat the import would check and only import new content and not overwrite that content that had already been migrated. However now we can only update once and there is an error if we complete subsequent uploads of data. This compounds the issue above with the content disappearing as we can only check the content works once. It would be great if the old method could come back where incremental imports from Slack could take place allowing for testing and gradual transition.

This was diagnosed as a Slack import issue through the following steps:

  1. Imported the slack export to 17th Dec (the last import before the problems began). The import worked fine, and there were no issues with channels or threads with disappearing content.
  2. Next we tried a full incremental import of slack data to Feb 1st. It complained of a duplicate record and failed to complete the import. That means incremental overlapping imports no longer work.

Steps to reproduce: As described above.

Expected behaviour: Incremental Slack imports don't fail.

Actual behavior: Mongo DB complained of a duplicate record and failed to replicate

Server Setup Information: Version of Rocket.Chat Server: 3.11.0 Operating System: Mac OS Deployment Method: Number of Running Instances: 1 DB Replicaset Oplog: NodeJS Version: 12.18.4 MongoDB Version: 4.2.12 Apps Engine version: 1.22.1

Client Setup Information Desktop App or Browser Version: Both Operating System: Mac OS Browsers tested: Chrome, Safari Additional context

Relevant logs: From server log I20210205-18:26:46.819(0) server.js:204 Slack Importer ➔ error MongoError: E11000 duplicate key error collection: rocketchat-cmty.rocketchat_message index: id dup key: { _id: "slack-C010RRW7DBK-1590048159-202600-file1" } at Function.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/error.js:57:12) at toError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:123:22) at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/common_functions.js:379:39 at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset.js:1204:22) at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:405:18 at processTicksAndRejections (internal/process/task_queues.js:79:11) { driver: true, index: 0, code: 11000, keyPattern: { _id: 1 }, keyValue: { _id: 'slack-C010RRW7DBK-1590048159-202600-file1' } }

close-issue-app[bot] commented 3 years ago

This issue was closed because it does not use our bug report issue template.

Please make sure to use it and fill it as much as you can so we can provide better and faster support.

The following sections must not be removed, or else the BOT will close it immediately again: