Emergency Breaking API Change

[jhgg]

The bulk_delete endpoint will no longer delete messages older than 2 weeks.

This was not a decision made lightly, as I've finally narrowed down the root cause of the recent API stability issues to people using bulk delete to purge channels with extensive message history, creating excessive table fragmentation within our cassandra cluster.

We'll be introducing a proper "purge channel" endpoint at some point in the future - which will be able to delete old messages without creating excessive table fragmentation (we already have a plan on how to do so).

For now, bulk delete requests will silently discard older messages when attempting to delete. The next time we roll APIs, the API endpoint will fail with a bad request, with error code: 50034 if you send it a message ID that is too old.

You can figure out the minimum allowed snowflake with the following code:

long((time.time() - 14 * 24 * 60 * 60) * 1000.0 - 1420070400000L) << 22

• [x] silently fail old message deletes
• [x] deploy APIs (which will add the error)
• [x] update docs.

[foxbot]

You guys are doing God's work, keep it up. 👍

[LaxSlash]

Good job tracking down the issue. 👍

[Waspyeh]

Awesome work guys

[galenguyer]

No messages can be deleted with a bot, even messages sent within the last hour.

EDIT: Issue fixed now

[jhgg]

When I pushed this out, I deleted a line of code that actually broke bulk deletes entirely. Whoops.

It's fixed now 👌

[AraHaan]

:man_facepalming: Well there goes my fun with bulk delete.

[shkkmo]

Is there a issue open for 'introducing a proper "purge channel" endpoint at some point in the future'? Even if this is just something for the UI so I can get rid of old stuff manually

[jhgg]

We have no plans. I recommend clone channel -> then delete the old one.

[shkkmo]

The documentation currently says "An endpoint will be added in the future to prune messages older than 2 weeks from a channel."

I don't see any api endpoint for cloning a channel, what are you talking about?

[MinnDevelopment]

You can create a channel with the same properties

[SomeoneCares11]

@minndevelopment but channel_id will change, so u need to edit your bots properties too

[MinnDevelopment]

Since the bot is performing the update that should not be an issue.

[shkkmo]

So the only solution to bulk delete old message is to completely recreate the channel and re-configure all integrations that use it? Won't this also remove any notification settings users have and break all invites and links to it? That's pretty unfortunate.

At the very least, the documentation should be updated to no longer indicate that a solution is coming for this.

[SomeoneCares11]

@shkkmo Time ago, you can't delete default channel, and everyone mass deleting messages - that why they said An endpoint will be added in the future to prune messages older than 2 weeks from a channel. At least now you can recreate that channel, but i agree with you it is not solution.

[macdja38]

Through a bot if your bot has the right permissions you could shift the webhooks to target the new channel before deleting the old one? That would probably solve a significant portion of your integration issues.

Still deletes invite links, user notification settings (I think)...

[davefrassoni]

Workaround for performance issues: add visible flag in messages and show only those with flag on. Then on low activity times perform background cleaning tasks.

[Kein]

What if I want to delete a few specific messages (let's say 100) of specific user on a specific channel older than 2 week? Judging by your suggestion above "delete and clone channel" - should I then ban the user with "prune message" option? This has a faint smell of hypocrisy and double-speak in the light of: https://blog.discordapp.com/the-future-of-bots-on-discord-4e6e050ab52e

[AEnterprise]

@Kein you can still delete them one by one, just not batched, it's a lot slower but still works

[Kein]

Yes, and get banned for "API abuse" because their rules and definition for this term so loose you can get banned for literally anything even if it is normal usage. Been there, done that.

[GEOEGII555]

It's 2022, there's still no endpoint

[Zoddo]

It's 2022, there are no endpoint

There is an endpoint in a way. Changes have been made to be able to delete the old default channels. So you can just clone and delete a channel to purge it.

[GEOEGII555]

It's 2022, there are no endpoint

There is an endpoint in a way. Changes have been made to be able to delete the old default channels. So you can just clone and delete a channel to purge it.

I mean endpoint for purging the channel, not for cloning/deleting channels

[Kein]

It's 2022, there are no endpoint

There is an endpoint in a way. Changes have been made to be able to delete the old default channels. So you can just clone and delete a channel to purge it.

That is not what this issue about, the discussion was a bout a bulk_delete of bunch of messages for valid usecases. Right now, you have to delete them 1 by 1 which means you subject to very strict API limitations and very easy bot ban (which are "always valid and final" and you cant even appeal despite Discord happily claiming they have specific community just for API users that topic is basically a taboo there)

[Zoddo]

Just so everyone understands why this limitation was added in the first place, there is an actual technical reason why Discord doesn't want you to prune large chunks of messages in channels (and can maybe even take actions against bots doing this as mentioned by Kein).

In distributed databases (like the one used by Discord), this generates a large amount of tombstone records, which can cause big performance issues (for the history, the pruning of a large guild's default channel in early 2017 -just before this issue was created-, is the root cause of API issues which lasted a few days -you can read a bit more about it in this blog post).

Although only Jake can confirm this, I'm fairly confident that the plan for the "proper purge channel endpoint" they wanted to introduce was anyway something close to deleting/recreating the channel (remember that people were pruning channels content mainly because the default channel was not deletable at that time).

[Kein]

(remember that people were pruning channels content mainly because the default channel was not deletable at that time).

Again, this fallacy being brought up just as an excuse. People will find a way to abuse anything. We arent talking here about channel deletion and recretation, we are talking about deleting big amount of messages in a bulk. For example, messages from particular bad actor throughout history of the server (ban option only allows to delete 24hrs long message) or multiple spam messages or just bunch of requested messages. There is plenty of absolutely legitimate and valid uses, yet every time this one specific abuse being used to explain "why we cant have nice things" as if like this is end of the world.

[Zoddo]

I'm not speaking of an abuse (I don't think anyone said that pruning a channel content was an abuse by itself), but an actual technical limitation.

Of course, putting limitations will always disturb valid use cases, and this has to be balanced. In this case, Discord has to balance between "deleting millions of messages can cause database issues" and "people need to be able to delete messages in bulk for a couple of reasons". Allowing to bulk delete messages posted in the last 15 days covers the most common use cases of the bulk delete endpoint while limiting the risk of causing performance issues. One of the very few cases that aren't covered by this decision is emptying a channel, and (from my pov) this is also the reason invoked by the majority of users complaining about this limitation. That's why this is often used as an example.

Now, I have a real question: How often do you need to delete messages older than 15 days in bulk?
I don't even remember having to need to do that, except for pruning the default channels (back when there weren't deletable).

[Kein]

How often do you need to delete messages older than 15 days in bulk?

If it is not as often as X/Y then it should not need to be available

A fallacy. Thousands of people delete their messages every day via Discord client, including purging very old ones. The """"damage"""" you are talking about is being done every day on a way bigger scale than a few hundred bots will ever do.