[Suggestion] Use webhooks to bypass rate limits

[NoNameLmao]

Instead of one bot sending messages, it could create webhooks which then would upload the files at speeds faster than one bot.

[ledlamp]

that would only double the rate in one channel as all webhooks in a channel are under the same rate limit. separate channels need to be used to circumvent the rate limit completely. a guild (max 500 channels) could be used as a data store instead of just one channel.

[forscht]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

[lunyaUwU]

would migration be possible?

[UnlimitedBytes]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

You need to use webhooks in order to get more speed. I tested it on my setup with one channel and can upload ~25 Mbit/s (a chunk in 2s on average sometimes delayed bc ratelimit) I tested it with 10 channels with the bot and still can upload ~25 Mbit/s I tested it with 20 channels over webhooks and can put up ~600 Mbit/s (the max. amount of webhooks you can create in a short span of time, without getting blocked for a very long time from creating webhooks)

Edit: Also if you use mutliple servers you can via the bot create 20 channels and webhooks per server without getting blocked I used this to fill up 5 server for a total of 100 channels. Now I can put up 2 Gbit/s ( which is my network limit :) )

[forscht]

@UnlimitedBytes , Can you create PR with your code?

[UnlimitedBytes]

@UnlimitedBytes , Can you create PR with your code? @forscht

Sadly I wrote a completely different implementation for myself which uses lots of libraries and multipart/form-data instead of direct binary data. So it's not possible to PR these changes as they're running on a completely different architecture and git repository.

I could however port it to DDrive will take some time tho I will prob. release the PR on the weekend as that's when I have time :)

[metenn]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

You need to use webhooks in order to get more speed. I tested it on my setup with one channel and can upload ~25 Mbit/s (a chunk in 2s on average sometimes delayed bc ratelimit) I tested it with 10 channels with the bot and still can upload ~25 Mbit/s I tested it with 20 channels over webhooks and can put up ~600 Mbit/s (the max. amount of webhooks you can create in a short span of time, without getting blocked for a very long time from creating webhooks)

Edit: Also if you use mutliple servers you can via the bot create 20 channels and webhooks per server without getting blocked I used this to fill up 5 server for a total of 100 channels. Now I can put up 2 Gbit/s ( which is my network limit :) )

How exactly does this work? Do you send multiple files in one webhook request (using multipart/form-data), multiple requests in parallel at the same time or multiple requests parallel to different channels/servers? I'm curious because while I got a speedup it was never as big as you mention in your post.

[UnlimitedBytes]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

You need to use webhooks in order to get more speed. I tested it on my setup with one channel and can upload ~25 Mbit/s (a chunk in 2s on average sometimes delayed bc ratelimit) I tested it with 10 channels with the bot and still can upload ~25 Mbit/s I tested it with 20 channels over webhooks and can put up ~600 Mbit/s (the max. amount of webhooks you can create in a short span of time, without getting blocked for a very long time from creating webhooks) Edit: Also if you use mutliple servers you can via the bot create 20 channels and webhooks per server without getting blocked I used this to fill up 5 server for a total of 100 channels. Now I can put up 2 Gbit/s ( which is my network limit :) )

How exactly does this work? Do you send multiple files in one webhook request (using multipart/form-data), multiple requests in parallel at the same time or multiple requests parallel to different channels/servers? I'm curious because while I got a speedup it was never as big as you mention in your post. @mattmeten

You send multiple parts at the same time in parallel. So for example you create 2 channels and add 20 webhooks per channel so you get 40 webhook urls in total. You then take the let's say 1 GB big file and split it into 40 chunks that are 8 MB big. You send all this 40 chunks at the same time, each to a different webhook. You then when a webhook finished it's 8 MB chunk stuff a new 8 MB chunk into this webhook. You constantly keep up the 40 connections in parallel until the file is finished and no more data is there to be uploaded. After that you have a ton of chunks uploaded with 40 chunks per webhook timing instead of 1 chunk per webhook timing. This process can be upscaled to as much as you want and as much as your processor / ram and network can handle.

[ledlamp]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

You need to use webhooks in order to get more speed. I tested it on my setup with one channel and can upload ~25 Mbit/s (a chunk in 2s on average sometimes delayed bc ratelimit) I tested it with 10 channels with the bot and still can upload ~25 Mbit/s I tested it with 20 channels over webhooks and can put up ~600 Mbit/s (the max. amount of webhooks you can create in a short span of time, without getting blocked for a very long time from creating webhooks) Edit: Also if you use mutliple servers you can via the bot create 20 channels and webhooks per server without getting blocked I used this to fill up 5 server for a total of 100 channels. Now I can put up 2 Gbit/s ( which is my network limit :) )

How exactly does this work? Do you send multiple files in one webhook request (using multipart/form-data), multiple requests in parallel at the same time or multiple requests parallel to different channels/servers? I'm curious because while I got a speedup it was never as big as you mention in your post. @mattmeten

You send multiple parts at the same time in parallel. So for example you create 2 channels and add 20 webhooks per channel so you get 40 webhook urls in total. You then take the let's say 1 GB big file and split it into 40 chunks that are 8 MB big. You send all this 40 chunks at the same time, each to a different webhook. You then when a webhook finished it's 8 MB chunk stuff a new 8 MB chunk into this webhook. You constantly keep up the 40 connections in parallel until the file is finished and no more data is there to be uploaded. After that you have a ton of chunks uploaded with 40 chunks per webhook timing instead of 1 chunk per webhook timing. This process can be upscaled to as much as you want and as much as your processor / ram and network can handle.

but all webhooks in a channel share the same rate limit ??? or does sending at the same time circumvent that somehow?

[UnlimitedBytes]

@NoNameLmao, @asdfisdffdg I already tried with using multiple channels but I couldn't see any major performance improvements. Still in upcoming release I will try to add multiple channels as option.

You need to use webhooks in order to get more speed. I tested it on my setup with one channel and can upload ~25 Mbit/s (a chunk in 2s on average sometimes delayed bc ratelimit) I tested it with 10 channels with the bot and still can upload ~25 Mbit/s I tested it with 20 channels over webhooks and can put up ~600 Mbit/s (the max. amount of webhooks you can create in a short span of time, without getting blocked for a very long time from creating webhooks) Edit: Also if you use mutliple servers you can via the bot create 20 channels and webhooks per server without getting blocked I used this to fill up 5 server for a total of 100 channels. Now I can put up 2 Gbit/s ( which is my network limit :) )

How exactly does this work? Do you send multiple files in one webhook request (using multipart/form-data), multiple requests in parallel at the same time or multiple requests parallel to different channels/servers? I'm curious because while I got a speedup it was never as big as you mention in your post. @mattmeten

You send multiple parts at the same time in parallel. So for example you create 2 channels and add 20 webhooks per channel so you get 40 webhook urls in total. You then take the let's say 1 GB big file and split it into 40 chunks that are 8 MB big. You send all this 40 chunks at the same time, each to a different webhook. You then when a webhook finished it's 8 MB chunk stuff a new 8 MB chunk into this webhook. You constantly keep up the 40 connections in parallel until the file is finished and no more data is there to be uploaded. After that you have a ton of chunks uploaded with 40 chunks per webhook timing instead of 1 chunk per webhook timing. This process can be upscaled to as much as you want and as much as your processor / ram and network can handle.

but all webhooks in a channel share the same rate limit ??? or does sending at the same time circumvent that somehow?

A sorry my brain is miss functional you add 1 webhook per channel but 20 channels :D

[ledlamp]

yes that's what i was saying. but the bot also has separate rate limit per channel, so you could just use the bot itself. or you could use both for double per channel. but it might be simpler to just make more channels.

[forscht]

This improvement has been added in version v4.0. I was able to upload 5GB of file under 85 seconds.