[Mutes] Re-joining guild is nullifying the `mute` if autorole exist

cool-aid-man commented 5 months ago

What Red version are you using?

3.5.7

Cog name

Mutes

Command name

mute

What did you expect to happen?

This came to our attention when a few users of our server recently were able to re-join the server with no muted - bypassing the mute persistent (when they were muted permanently beforehand)

So I tested it and it turns out, you can actually bypass mute by rejoining a server, in the presence of ANY autorole/joinrole cog - In a system where you set certain role(s) as joinrole which gets added to the user upon joining the server.

So, to answer the question, The expected behavior would be - Persistent Mute over guild Re-Join. (I know Red already does but it's failing, details are as follows - )

What actually happened?

Findings from the test:

It doesn't matter which autorole cog you're using (I tested with 3rd party autoroler cog by dav & autorole by lemon - both verified), when there's autorole + mute, autorole gets 1st priority nullifying the mute on member_join.
- Both the Mutes & autorole cogs are loaded on the same red instance. There's NO other interference from ANY other bots whatsoever (I made sure of it before testing)
It's even happening when the autorole system is present on a completely different 3rd party bot that has no connection with red & the muting is done with red instance. Which is really weird. Same result here - roles that are set as joinrole/autorole gets added to the user but not the mute role.
Here's the issue: This could be an underlying red or, dpy issue as this is happening on multiple occasions not entirely restricted to Mutes cog. (For example, if you execute say, 3 commands in one go, with one invoke - then chances are any of the 2 commands won't even get executed in the first place 5 out of 10 times - but this requires its own issue).
Audit log (server audit log) says red did add the role. :point_down:
But , in reality, the mute role got removed from the user - logs (from extendedmodlog - by trusty ) support that statement - "The mute role got removed by the bot." Now yes it's true that extendedmodlog guesses the events but still as there's nothing in the server audit log that says otherwise.
- Here in the SS ^ the 3k role was set as autorole/joinrole through the 3rd party autoroler cog (by dav) | The muted was set on the same bot as well.
There's NOTHING else in the guild_audit log that says the mute role was removed in any other way.

How can we reproduce this error?

Test 1 - `Mutes` and `AutoRole` Cogs are on the same instance

Load Mutes cog & set a muted role + logs.
Install ANY autorole/joinrole cog and set a test role to be added as autorole - cog that adds a role as soon as a human joins a guild.
Test it in a 3k+ server.
Mute a test-account/user for any duration - enough so it can cover the test duration.
Leave the guild after that - from that user account.
Re-join as the same user. You will notice the user doesn't have the mute role anymore, however they will have the autorole.

You can install extendedmodlog by trusty (verified) to check further.

Test 2 - Mutes is on the Red instance, Autorole is set to any 3rd party bot.
1. Load Mutes cog & set a muted role + logs.
2. Use any 3rd party bot such as carl bot (235148962103951360) - it has a autorole module, set any test role as autorole/joinrole.
3. Mute a test-account/user for any duration with the Red instance - enough so it can cover the test duration.
4. Leave the guild after that - from that user account.
5. Re-join as the same user. You should be able to notice the same result - that the user doesn't have the mute role anymore, however They will have the autorole.

Anything else?

I've tried to fill this issue, with as much info as possible, sry if it is still not enough. If it's not enough or requires more testing, then I'd love to give a hand. Please ping me if you need more info or testing.

Btw I believe red current versions are not the reason for this issue, since these are minor bump 3.5.x and this could be an overlong persisting issue coming from 3.5.0 dpy 2.0 causing not just mute persistent problem but also multiple role add & remove issue.

Flame442 commented 5 months ago

I can't reproduce test case 1, my bot properly gives back both roles. Though I didn't do it in a 3k+ server since I don't have access to a server that large where I can easily test something like this. Please try this on a fresh instance and report back if you still experience this issue, to rule out any potential for other cogs to be conflicting in some way (ie exclusive roles).

cool-aid-man commented 5 months ago

Sure, Here are the results.

Test case 2 - `Mutes` is on the Red instance, `Autorole` is set to any 3rd party bot.

Made a new instance for a fresh start, with a New test-mute role to avoid conflicting with other roles. (Testing Server is 3k+ here as well).
For the autorole I used two different 3rd party bots; one that I had mentioned earlier carl bot (235148962103951360) another one is a red instance different than the bot I'm muting with ofc.
Upon re-joining after getting muted, the server-audit log says, the test bot, added the mute role. The 3rd part bot also says they added the autorole.
In Both cases the role that is set as autorole, gets added first (and I think this is purely because of how fast the 3rd party bots/instances are - in terms of adding autorole), and whenever the autorole gets added first, the mute role gets removed. 👇 - There are No other influences whatsoever.
In reality: The user got unmuted. 👆

6/6 Times - same results. i.e. Muted role getting removed upon re-joining.

Test case 1 - Mutes & Autorole are on the same New Red instance

Everything is the same as test case 1. (3k+ test server as well) Results:
4/6 Times upon rejoining the mute role got removed & autorole got added alone. 2/6 times the mute role got added ONLY but no autorole.

And I think this is also because this new test instance isn't fast enough to add the autorole first (unlike the # test-case-1 where the bots and the red instance were super-fast adding the autorole ).

Tl;dr: Whenever a role is getting added first (upon joining the guild), the mute role somehow gets removed from the user. Doesn't matter which bot you're using, if its fast enough adding role(s), then the mute role will get removed leading to nullifying the process overall.

I don't have any other autorole or exclusive role cogs in any of the instances, as a precaution I had quarantined ALL the other bots before doing the test. (Had done it earlier as well)

cool-aid-man commented 5 months ago

One more thing, Since test servers (with <100 members) are not ideal for testing - due to several reasons and since it's kind of hard to get your hands on 3k+ servers, what we can do is - we can provide a private space for ANY ORG/staff member who may wish to come in, to test this out in our server, yes with their own bots to pinpoint the root of this issue and make it better for everyone else. I think from our end, we as a server, will be able to provide as much info and help as is needed to solve it.

@Flame442 Flame, if you wish you can come to our server with your own bot .gg/thestarship that's The Starship (this is where the test results are from - and for the "13-year-olds" no this is Not an advertisement, this is a mere attempt to find what is causing it). We will provide you with all you need to see this phenomenon happening with your own eyes and may be able to find the reason behind it.

Anyone is welcome, just ping me if any-one decides to come in so that I can provide you the tools that you need for testing.

Servers under 100 might not be able to showcase this issue 10/10 times but I believe any server 1000+ with lots of roles, should be able to recreate the mentioned issues (both test cases: 1 and 2).

Flame442 commented 5 months ago

While it seems like this issue is possible to reproduce given the specific server environment you have, it does seem like it requires some stars to align in order to happen. I'm also not entirely convinced we've tracked down the source of the issue, but we at least found that adding a sleep before adding the autoroles or moving the autorole to the default onboarding roles will fix this issue. I think it's a bit too niche and a bit too risky to fix in Mutes, since the only way to do it would be to delay adding the mute role on rejoin. For that reason, I'm going to consider this an issue that would be better fixed in other ways, namely in the 3rd party autorole cogs.

Cog-Creators / Red-DiscordBot