serenity-rs / songbird

An async Rust library for the Discord voice API
ISC License
391 stars 111 forks source link

Possible memory leak in Serenity 0.12 and Songbird 0.4. #205

Open abokado0107 opened 10 months ago

abokado0107 commented 10 months ago

Songbird version: (version) 0.4.0-rc

Rust version (rustc -V): (version) 1.74.0

Serenity/Twilight version: (version) Serenity 0.12.0-rc

Description: When a large number of small audio files are played back, CPU utilization increases and memory usage explodes after a certain period of time. image

Steps to reproduce: Rust installation is omitted.

$ sudo apt update && sudo apt install libopus-dev build-essential autoconf automake libtool m4 pkg-config libssl-dev -y
$ git clone https://github.com/abokado0107/tts-memory-leak
$ cd tts-memory-leak
$ cargo build --release
  1. Build the verification repository.
  2. Enter token in .env and run it.
  3. Type !join on Discord to invite the BOT to the voice channel.
  4. After a few days, the memory usage will explode and crash in a short time. It crashes faster if you have it connected to several VCs at the same time.

Verification environment CPU: ARM Ampere CPU: 2 core RAM: 4 GB OS: Ubuntu 20.04 IaaS: Oracle Cloud Infrastructure Repositories for verification: https://github.com/abokado0107/tts-memory-leak

FelixMcFelix commented 10 months ago

I've been running my own bot continuously for a day or so which also plays back many small audio files; I'm not seeing my CPU or memory substantially increase over time in a single server. I'll keep you posted if anything changes on that front.

abokado0107 commented 10 months ago

I don't know if this is directly related to this issue, but after several days of running my application, I am accumulating tasks that never finish. This image is displayed by tokio-console. image

I am not familiar with the structure of serenity or songbird, but it seems that tasks generated by songbird's task.rs and serenity's tokio.rs remain relatively well.

FelixMcFelix commented 10 months ago

I need a little more context; one of each driver/scheduler/task.rs:112 is expected for every Call without live audio playing, If there's a big difference between those two quantities then that would indicate a task leak. If you're joining/creating Calls but not removing, this would be expected, otherwise I'm very interested.

abokado0107 commented 10 months ago

Thanks for letting me know. I checked in the production environment and Call was able to remove properly. I output the following counts to the debug log and the call was increasing and decreasing properly.

if let Some(manager) = songbird::get(ctx).await {
    let call_count = manager.iter().count();
}

image

As for the scale of the application, the number of simultaneous VC connections at 30k guilds is about 2k at peak time.