Rocketchat import issue

[manc74]

Attempting a new deployment of Zulip and importing data from Rocket chat. Following the instructions im getting to the data conversion and getting this error:

Converting Data ...
Traceback (most recent call last):
  File "/home/zulip/deployments/2023-05-15-08-51-03/./manage.py", line 151, in <module>
    execute_from_command_line(sys.argv)
  File "/home/zulip/deployments/2023-05-15-08-51-03/./manage.py", line 116, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/management/commands/convert_rocketchat_data.py", line 45, in handle
    do_convert_data(rocketchat_data_dir=data_dir, output_dir=output_dir)
  File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/data_import/rocketchat.py", line 1074, in do_convert_data
    process_users(
  File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/data_import/rocketchat.py", line 96, in process_users
    delivery_email = user_dict["emails"][0]["address"]
KeyError: 'emails'

Any ideas?

[manc74]

I've managed to "resolve" this - it was caused by a user with a "+" in the email address causing the issue. Ive removed that bot user now and the process runs,

[zulipbot]

Hello @zulip/server-misc members, this issue was labeled with the "area: export/import" label, so you may want to check it out!

[alexmv]

To make sure I'm understanding right -- you removed the bot user in RocketChat and re-exported, then ran the conversion on that new export, and it succeeded?

[manc74]

Yes that's correct.

[alexmv]

OK! If you have the old export file still around, can you try applying this diff and re-running the conversion on it?

diff --git zerver/data_import/rocketchat.py zerver/data_import/rocketchat.py
index 5c4d32e184..0b21189449 100644
--- zerver/data_import/rocketchat.py
+++ zerver/data_import/rocketchat.py
@@ -71,6 +71,7 @@ def process_users(
         # "user": This is a regular user of the system.
         # "bot": A special user types for bots.
         # "unknown": This usually represents a livechat guest.
+        logging.info("%r", user_dict)
         if user_dict["type"] != "user":
             is_active = False

Running ./manage.py convert_rocketchat_data will not affect any data in your current Zulip instance, and the patch is also safe to apply to a running Zulip instance -- the serving codepaths don't use that file at all.

The output should be a bunch of user data -- we're interested in the last one output. You can redact anything user-specific in it -- we're interested in the following keys:

• type (exact value is important)
• username (mostly just presence or absence)
• emails (mostly just presence or absence)

[manc74]

Sure, I'll give it a go in the morning and let you know the results! Thanks.

[manc74]

Actually, just ran it now as I had a local copy of the dump - this is the output of the last record (redacted) and also the traceback:

2023-05-16 19:38:05.637 INFO [] {'_id': 'RzcsZYNmZMFatF95Y', 'createdAt': datetime.datetime(2019, 9, 25, 12, 12, 4, 217000, tzinfo=<bson.tz_util.FixedOffset object at 0x7f7546822da0>), 'services': {'password': {'bcrypt': $2b$10$cEQxxxxxxxxxxxxxxxSd8Iu27i5t0XyFW'}}, 'username': 'redactedusername', 'type': 'user', 'status': 'offline', 'active': True, '_updatedAt': datetime.datetime(2021, 1, 28, 17, 43, 38, 771000, tzinfo=<bson.tz_util.FixedOffset object at 0x7f7 546822da0>), 'roles': ['bot'], 'utcOffset': -7, 'name': 'redactedusername', 'importIds': ['UNQLBPS2H', 'BNBSVGNM8'], 'rooms': []} redactedusername Traceback (most recent call last): File "/home/zulip/deployments/2023-05-15-08-51-03/./manage.py", line 151, in execute_from_command_line(sys.argv) File "/home/zulip/deployments/2023-05-15-08-51-03/./manage.py", line 116, in execute_from_command_line utility.execute() File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, *cmd_options) File "/srv/zulip-venv-cache/f3e3bffb2506c95b60d086c5350e262ba5380c78/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(args, **options) File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/management/commands/convert_rocketchat_data.py", line 45, in handle do_convert_data(rocketchat_data_dir=data_dir, output_dir=output_dir) File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/data_import/rocketchat.py", line 1076, in do_convert_data process_users( File "/home/zulip/deployments/2023-05-15-08-51-03/zerver/data_import/rocketchat.py", line 98, in process_users delivery_email = user_dict["emails"][0]["address"] KeyError: 'emails'

[alexmv]

Thanks for that! I've pushed a PR to resolve this -- though I'll admit I'm a bit confused about RocketChat's data model which permits users without email addresses.

[alexmv]

I've pushed another commit to key off of the roles: ['bot'] to treat that "user" as a bot.

[manc74]

No, the user did have an email address, but it was in the format of "bot+name@domain.com" - I suspect its the "+" plus addressing in the email address that was confusing either the export from Rocketchat or the import to Zulip.

[alexmv]

Must have been the export, since it's not in the logline you showed.