Shard ID None heartbeat blocked

[benishs]

Thanks for your work on this! I've tried a few slack to discord tools and this is the only one I could actually get working.

I got part way through an import, then it stalled and I started getting this (repeats every 10 seconds):

2022-10-08 09:31:45 WARNING discord.gateway Shard ID None heartbeat blocked for more than 35830 seconds. Loop thread traceback (most recent call last): File "/Users/benishs/opt/anaconda3/bin/slack-to-discord", line 8, in <module> sys.exit(main()) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 584, in main run_import( File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 563, in run_import client.run(token, reconnect=False, log_handler=None) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/discord/client.py", line 828, in run asyncio.run(runner()) File "/Users/benishs/opt/anaconda3/lib/python3.8/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/Users/benishs/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete self.run_forever() File "/Users/benishs/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 570, in run_forever self._run_once() File "/Users/benishs/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once handle._run() File "/Users/benishs/opt/anaconda3/lib/python3.8/asyncio/events.py", line 81, in _run self._context.run(self._callback, *self._args) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/discord/client.py", line 409, in _run_event await coro(*args, **kwargs) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 420, in on_ready await self._run_import(g) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 518, in _run_import sent = await self._send_slack_msg(ch_send, msg) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 442, in _send_slack_msg for attempt in file_upload_attempts(data): File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 368, in file_upload_attempts fp=io.BytesIO(urllib.request.urlopen(url).read()), File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 525, in open response = self._open(req, data) File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 542, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(*args) File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 1393, in https_open return self.do_open(http.client.HTTPSConnection, req, File "/Users/benishs/opt/anaconda3/lib/python3.8/urllib/request.py", line 1354, in do_open r = h.getresponse() File "/Users/benishs/opt/anaconda3/lib/python3.8/http/client.py", line 1332, in getresponse response.begin() File "/Users/benishs/opt/anaconda3/lib/python3.8/http/client.py", line 303, in begin version, status, reason = self._read_status() File "/Users/benishs/opt/anaconda3/lib/python3.8/http/client.py", line 264, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/Users/benishs/opt/anaconda3/lib/python3.8/socket.py", line 669, in readinto return self._sock.recv_into(b) File "/Users/benishs/opt/anaconda3/lib/python3.8/ssl.py", line 1241, in recv_into return self.read(nbytes, buffer) File "/Users/benishs/opt/anaconda3/lib/python3.8/ssl.py", line 1099, in read return self._sslobj.read(len, buffer)

Is there anything I can do to make it recover and continue the import? (It's still going, at the moment.)

If I re-run slack-to-discord will it ignore the channels / messages it has already imported, or do I need to start over? When I first tried to run slack-to-discord I attempted to remove some unneeded channel directories from the Slack export archive (unzipped, removed some directories, re-zipped), but slack-to-discord wouldn't start – an error about not finding the Users file. It would only work with the original exported .zip from Slack. So removing the already imported channels from that export doesn't seem to be an option.

Anything I can do to prevent this from happening again next time I run it?

Thanks,

Scott

[pR0Ps]

Ah, this is happening because the program blocks while it's downloading a file from Slack so it can upload it to Discord. This causes the Discord session to emit warnings because it can't properly keep the session alive.

I've already been working on a fix for this that streams the data from Slack to Discord instead of fully downloading it before starting the upload. Should be able to push something soon.

If you run it again, it will start from the beginning so you'll want to delete all the channels it did import from Discord before you run it again.

It should also work to delete the channel data from the zip file but might require tweaking some of the other files so they don't reference the channel anymore - I don't remember offhand. I typically just deleted everything and ran the full import again if I hit issues.

[benishs]

Thanks for the quick reply!

I again tried deleting some channels from the archive, because it took awhile to get through the stuff it has already done, but am getting the same error as before – in case that's useful:

2022-10-08 10:30:52 INFO slack_to_discord Logging the bot into Discord Traceback (most recent call last): File "/Users/benishs/opt/anaconda3/bin/slack-to-discord", line 8, in <module> sys.exit(main()) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 584, in main run_import( File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 562, in run_import client = SlackImportClient(data_dir=t, **kwargs) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 398, in __init__ self._users = slack_usermap(data_dir, real_names=real_names) File "/Users/benishs/opt/anaconda3/lib/python3.8/site-packages/slack_to_discord.py", line 102, in slack_usermap with open(os.path.join(d, "users.json"), 'rb') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/q8/n_7347m91r9dhc1m_lh271180000gn/T/tmpnr4ij0hy/users.json'

It seems to think users.json doesn't exist? But it definitely exists in that .zip, and I didn't touch that users.json file when I removed some of the channels. Even just removing 1 channel (that was only posted to by a bot) gives this error. ¯_(ツ)_/¯

If the easiest solution is to just start over, I can do that.

Thank you!

[benishs]

So, I edited the contents of the .zip without extracting it, using an old app called Zipster, and it seems to be working now with the truncated data.

Not surprisingly, it gave a bunch of errors about the missing channels when it started, e.g. Data for channel '#general' not found in export but then proceeded to start on the channels that still exist in the archive.

So, something about unzipping and then re-zipping the archive was causing the issue? That makes no sense to me, but I know very little about this stuff. FWIW, I was previously just using the built in unzip / zip tools in the Finder (macOS Monterey 12.6). Anyway, maybe that helps someone else out someday!

[pR0Ps]

It could be that when you recompressed it the files were added to the zip in a subfolder (ie. files were added to the zip as /folder_you_extracted_to/path/to/file.txt instead of just /path/to/file.txt).

I've pushed a new commit that should fix your issue with file uploads (the cause of the stall and warnings above). Feel free to re-open if you encounter the issue again.

[pR0Ps]

Reverted the commit as it had some issues I didn't catch when testing