search

terminal-discord / weechat-discord

Weechat plugin for Discord support - https://weechat.org/ https://discord.com/
MIT License
175 stars 15 forks source link

Panic in member list #101

Open Edremon opened 2 years ago

Edremon commented 2 years ago

I have been using weechat-discord (mk3 branch) in the last few days and after some time, around 1-2 hours, it panic, here what's in printed in weechat:

Panic in the main Weechat thread: panicked at 'removal index (is 99) should be < len (is 99)', src/twilight_utils/member_list.rs:100:31
discord: event loop failed, stopping...
discord: event loop failed, stopping...
discord: event loop failed, stopping...

Here what I have in tracing in a normal session right after logging in, I don't know if all those errors are relevant:

ERROR twilight_gateway::shard::json: invalid JSON: {"t":"THREAD_LIST_SYNC","s":43,"op":0,"d":{"threads":[{"type":11,"thread_metadata":{"locked":false,"invitable":true,"create_timestamp":null,"auto_archive_duration":10080,"archived":false,"archive_timestamp":"redacted"},"rate_limit_per_user":0,"parent_id":"redacted","owner_id":"redacted","name":"redacted","message_count":50,"member_ids_preview":["redacted","redacted","redacted","redacted"],"member_count":4,"last_pin_timestamp":"redacted","last_message_id":"redacted","id":"redacted","guild_id":"redacted","flags":0}],"most_recent_messages":[{"type":0,"tts":false,"timestamp":"redacted","referenced_message":null,"pinned":false,"nonce":"redacted","mentions":[],"mention_roles":[],"mention_everyone":false,"is_thread_dispatch":true,"id":"redacted","flags":0,"embeds":[],"edited_timestamp":null,"content":"redacted","components":[],"channel_id":"redacted","author":{"username":"redacted","public_flags":512,"id":"redacted","discriminator":"0500","avatar_decoration":null,"avatar":"redacted"},"attachments":[]}],"guild_id":"redacted"}}
ERROR twilight_gateway::shard::processor::r#impl: processing incoming event failed: ProcessError { kind: ParsingPayload, source: Some(GatewayEventParsingError { source: Some(Error("missing field `members`", line: 1, column: 1260)), kind: Deserializing }) } shard_id=0 shard_total=1
ERROR weecord::hooks::signals: Error acking channel history: failed to deserialize response body guild_id=Some(Id<GuildMarker>(redacted)) channel_id=Id<ChannelMarker>(redacted)

The first error seem to be about an active thread (that I haven't joined). The third one seem to be about a channel where I have unread messages.

I will update issue if I get more relevant logs when that error happen again.

Noskcaj19 commented 2 years ago

Mm, that's interesting.

The first two lines are due to some recent change to THREAD_LIST_SYNC, I'll have to look into that, and third means they changed the "mark as read" api, which I will also have to look into.

Neither of those should be related to the panic though. I'm not positive what could cause that panic, (very possibly an api change), however I suspect that another possible option would be if your client disconnected momentarily I suspect that it would be possible for the member list to become desynchronized between the client and server and then the panic might occur. I will try and see if I can replicate the error, and look into how to just reset (or maybe disable) the member list when something like this happens.

Also, while looking through the code I found a typo (which I fixed in 762a92a), although I don't believe it could cause the panic you encountered.

Edremon commented 2 years ago

The panic didn't happen today, when it happened multiple times yesterday, don't have any idea what is different, connected to the same exact channels.

Concerning the errors that I posted in my initial bug report, they don't seem to be related, I got the ack one a lot but that just seem seem to make reading state a bit weird.

Meanwhile, I also got new errors that seem to be more related: paste since it's quite big. It's always the same with the same two users in it being spammed quite frequently (currently 300k lines in buffer).

Edremon commented 2 years ago

Happened again, I don't see anything special in traces.

In weechat core:

hh:26:38    =!= Panic in the main Weechat thread: panicked at 'removal index (is 99) should be < len (is 99)', src/twilight_utils/member_list.rs:99:31
hh:26:38        discord: event loop failed, stopping...
hh:26:41        discord: event loop failed, stopping...
hh:26:44        discord: event loop failed, stopping...
hh:26:44        discord: event loop failed, stopping...
hh:26:44        discord: event loop failed, stopping...
hh:26:44        discord: event loop failed, stopping...
hh:26:45        discord: event loop failed, stopping...
hh:26:45        discord: event loop failed, stopping...
hh:26:46        discord: event loop failed, stopping...
hh:26:46        discord: event loop failed, stopping...
hh:26:46        discord: event loop failed, stopping...
hh:26:46        discord: event loop failed, stopping...

In weecord log:

hh:26:38        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:41        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:44        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:44        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:44        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:44        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:45        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:45        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:46        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:46        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:46        ERROR weecord::discord::discord_connection: Event loop failed: channel closed
hh:26:46        ERROR weecord::discord::discord_connection: Event loop failed: channel closed

In the 30 minutes before panic, it only had a few errors related to ack.

Edremon commented 2 years ago

It happened again without any of the Nick list in an invalid state error, so that might also be unrelated.

Noskcaj19 commented 2 years ago

The Nick list in an invalid state error might not be required for the panic to occur, but it's a very strong indication that either something major has changed in the API or one of the updates was missed.

To try and narrow down the cause:

I'll push a change (7740490) that should prevent the panic from occurring if you just want it to stop crashing, however the panic is there because once this happens there is no way to correct the nicklist state and the nicklist will be incorrect (users will get other users statuses) until weechat is restarted.

The only real way to trace this down would be to recompile with unlimited-logging enabled, add twilight_gateway::shard::processor=trace to the tracing directives. then the log file can be filtered to include only GUILD_MEMBER_LIST_UPDATE events. From there it would be possible to reconstruct the series of events that lead to the panic. Not easy though, and I still haven't been able to reproduce, although I will keep trying.

Edremon commented 2 years ago

Can you double check if there are any Failed to add group "{}" to nicklist errors? That could lead to this panic happening.

No Failed to add group "{}" to nicklist errors at all.

Do you have a very reliable internet connection, or is it possible that the client disconnected briefly?

AFAIK my network connection is reliable.

Are you in any very large group chats (100 users or more?)

Some have more than 10k members.

Were you actively using weecord? Specifically adding or removing channels or servers? Or was it left completely idle with no channels or guilds added or removed since weechat was opened?

Hmm most of the time, I'm on some irc channel when that happen. In those session, I have never added or removed channels/guild, only use autoconnect.