search

webrtc-rs / webrtc

A pure Rust implementation of WebRTC
https://webrtc.rs
Apache License 2.0
4.04k stars 359 forks source link

Minor Memory Leak Issue for each successful peerconnection #406

Open shiqifeng2000 opened 1 year ago

shiqifeng2000 commented 1 year ago

hi

I am trying to build an app using actix-web as an ice server, ffmpeg as video/video sample feeder and this repo as a proxy.

But after I put them together I found that each peer connection will left some few memory in the system process, after a lot of heavy visits test, the process will be killed by Linux system and quit.

At first, I found some udp connection failed to get closed after peer connection closed, which can be found from bugfix-Udp connection not close (reopen #174) #195 #202

But after it was fixed and the repo version got to 0.6, I found the memory issue is still there, each time when a peer connected a portion of 0.1% of my system memory is consumed, I guess it is likely to be some threading data issue.

this is the outcome for initial state

%Cpu(s): 16.6 us,  4.1 sy,  0.0 ni, 79.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7877.7 total,    117.5 free,   6234.4 used,   1525.8 buff/cache
MiB Swap:   7813.0 total,   4300.8 free,   3512.2 used.   1207.2 avail Mem 

 PID      USER      PR  NI    VIRT        RES    SHR         %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                               
 93263  robin        20   0     2216484  25620  20060 S  19.9       0.3           0:55.53 vccplayer

And here's a few connections from web 图片

this is what I got after 5 connection is released

%Cpu(s):  9.1 us,  5.6 sy,  0.0 ni, 85.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7877.7 total,    178.3 free,   6186.3 used,   1513.1 buff/cache
MiB Swap:   7813.0 total,   4271.2 free,   3541.8 used.   1244.3 avail Mem 

 PID     USER      PR  NI    VIRT        RES     SHR       %CPU  %MEM     TIME+ COMMAND                                                                                                                             
 93263 robin        20   0     2221880  64172  32396 S  26.7     0.8            2:05.10 vccplayer

no sockets left 图片

Your see, the mem is growing, even after all udp sockets are released. Ffmpeg, however, is likely to be another reason to leak memory, so I removed ffmpeg from my code, copying the code from example play-from-disk-h264 to track the issue. Still no luck. Maybe it's because of the webrtc repo code.

To prove my point, I use this repo example play-from-disk-h264 as test subject, with an h264 and an ogg file split from a mp4 file, valgrind, --leak-check=full to test if any data is still reachable.

This is what I got

Peer Connection State has changed: closed
==77483== 
==77483== HEAP SUMMARY:
==77483==     in use at exit: 10,536 bytes in 73 blocks
==77483==   total heap usage: 137,624 allocs, 137,551 frees, 5,498,181 bytes allocated
==77483== 
==77483== 32 bytes in 1 blocks are still reachable in loss record 1 of 9
==77483==    at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483==    by 0x12BC8CB: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x12BC956: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x12BD7E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x12BC82C: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x12B7FDE: parking_lot_core::parking_lot::HashTable::new (boxed.rs:215)
==77483==    by 0x12B84D7: parking_lot_core::parking_lot::create_hashtable (parking_lot.rs:237)
==77483==    by 0x12B848F: parking_lot_core::parking_lot::get_hashtable (parking_lot.rs:225)
==77483==    by 0x12B85D3: parking_lot_core::parking_lot::grow_hashtable (parking_lot.rs:267)
==77483==    by 0x12B8314: parking_lot_core::parking_lot::ThreadData::new (parking_lot.rs:180)
==77483==    by 0x12B8DCD: parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__init (parking_lot.rs:202)
==77483==    by 0x12B8E30: parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{{closure}} (local.rs:353)
==77483== 
==77483== 128 bytes in 1 blocks are still reachable in loss record 2 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483==    by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483==    by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483==    by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483==    by 0x6FDE73: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483== 
==77483== 128 bytes in 1 blocks are still reachable in loss record 3 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483==    by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483==    by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483==    by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483==    by 0x10DB18D: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483== 
==77483== 128 bytes in 1 blocks are still reachable in loss record 4 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483==    by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483==    by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483==    by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483==    by 0xAF36B3: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483== 
==77483== 128 bytes in 1 blocks are still reachable in loss record 5 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483==    by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483==    by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483==    by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483==    by 0xAF38ED: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483== 
==77483== 128 bytes in 1 blocks are still reachable in loss record 6 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x10E6D8B: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x10E6E16: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x10E7119: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x10E6CEC: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x10E6BF2: <alloc::boxed::Box<T> as core::default::Default>::default (boxed.rs:1259)
==77483==    by 0x10E5889: arc_swap::debt::list::Node::get::{{closure}} (list.rs:164)
==77483==    by 0x10E661B: core::option::Option<T>::unwrap_or_else (option.rs:825)
==77483==    by 0x10E57C9: arc_swap::debt::list::Node::get (list.rs:148)
==77483==    by 0xAF42DD: arc_swap::debt::list::LocalNode::with::{{closure}} (list.rs:222)
==77483== 
==77483== 1,024 bytes in 1 blocks are still reachable in loss record 7 of 9
==77483==    at 0x484D7E3: memalign (vg_replace_malloc.c:1531)
==77483==    by 0x484D8FF: posix_memalign (vg_replace_malloc.c:1703)
==77483==    by 0x1480BF9: aligned_malloc (alloc.rs:97)
==77483==    by 0x1480BF9: alloc (alloc.rs:22)
==77483==    by 0x1480BF9: __rdl_alloc (alloc.rs:378)
==77483==    by 0x12BC8CB: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x12BC956: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x12BD7E9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x12BDE20: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:185)
==77483==    by 0x12BE43C: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:131)
==77483==    by 0x12BA7B3: alloc::vec::Vec<T,A>::with_capacity_in (mod.rs:641)
==77483==    by 0x12BA4F6: alloc::vec::Vec<T>::with_capacity (mod.rs:483)
==77483==    by 0x12B7DEC: parking_lot_core::parking_lot::HashTable::new (parking_lot.rs:75)
==77483==    by 0x12B84D7: parking_lot_core::parking_lot::create_hashtable (parking_lot.rs:237)
==77483== 
==77483== 2,080 bytes in 1 blocks are still reachable in loss record 8 of 9
==77483==    at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483==    by 0x12144DB: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x1214566: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x1215FE9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x11E0610: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:185)
==77483==    by 0x11E4DCC: alloc::raw_vec::RawVec<T,A>::with_capacity_in (raw_vec.rs:131)
==77483==    by 0x1197E63: alloc::vec::Vec<T,A>::with_capacity_in (mod.rs:641)
==77483==    by 0x1196B96: alloc::vec::Vec<T>::with_capacity (mod.rs:483)
==77483==    by 0x118E5BC: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter (spec_from_iter_nested.rs:54)
==77483==    by 0x119BB04: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter (spec_from_iter.rs:33)
==77483==    by 0x119B5CC: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter (mod.rs:2645)
==77483==    by 0x11F9BC4: core::iter::traits::iterator::Iterator::collect (iterator.rs:1792)
==77483== 
==77483== 6,760 bytes in 65 blocks are still reachable in loss record 9 of 9
==77483==    at 0x484884F: malloc (vg_replace_malloc.c:393)
==77483==    by 0x12144DB: alloc::alloc::alloc (alloc.rs:89)
==77483==    by 0x1214566: alloc::alloc::Global::alloc_impl (alloc.rs:171)
==77483==    by 0x1215FE9: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:231)
==77483==    by 0x121443C: alloc::alloc::exchange_malloc (alloc.rs:320)
==77483==    by 0x11BA913: alloc::sync::Arc<T>::new (boxed.rs:215)
==77483==    by 0x122FA79: tokio::sync::watch::channel (watch.rs:319)
==77483==    by 0x122E2CD: <tokio::signal::registry::EventInfo as core::default::Default>::default (registry.rs:22)
==77483==    by 0x11F8768: <tokio::signal::unix::SignalInfo as core::default::Default>::default (unix.rs:227)
==77483==    by 0x11F85F9: tokio::signal::unix::<impl tokio::signal::registry::Init for alloc::vec::Vec<tokio::signal::unix::SignalInfo>>::init::{{closure}} (unix.rs:33)
==77483==    by 0x11FA010: core::iter::adapters::map::map_fold::{{closure}} (map.rs:84)
==77483==    by 0x11F9B6A: core::iter::range::<impl core::iter::traits::iterator::Iterator for core::ops::range::RangeInclusive<A>>::fold::ok::{{closure}} (range.rs:1161)
==77483== 
==77483== LEAK SUMMARY:
==77483==    definitely lost: 0 bytes in 0 blocks
==77483==    indirectly lost: 0 bytes in 0 blocks
==77483==      possibly lost: 0 bytes in 0 blocks
==77483==    still reachable: 10,536 bytes in 73 blocks
==77483==         suppressed: 0 bytes in 0 blocks
==77483== 
==77483== For lists of detected and suppressed errors, rerun with: -s
==77483== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Is this because of the tokio runtime? Or tokio task not released in your code? I'm not sure, could you examine your code to see if there's leaking issue? Thanks

kraigher commented 1 year ago

I have also noticed memory leaks for each peer connection. I would suspect that there are cyclic references created by the frequent use of Arc in this project.

This project is to a large degree a 1:1 clone of the Pion server written in Go and thus shares the same software architecture. Go has a garbage collector that probably (my guess) can detect dead islands of cyclic references unlike in Rust where an island of cyclic Arc references will never be freed.

lf94 commented 10 months ago

From what I understand this will be a quite a large issue for long running services that rely on WebRTC. Is there a reason the issue isn't higher priority? Nice breakdown @shiqifeng2000