search

rust-lang / futures-rs

Zero-cost asynchronous programming in Rust
https://rust-lang.github.io/futures-rs/
Apache License 2.0
5.43k stars 628 forks source link

FuturesUnordered produces invalid reads and writes under Valgrind #2781

Open dignifiedquire opened 1 year ago

dignifiedquire commented 1 year ago

We have been debugging spurious issues with segfaults for a while, and while running our code through valgrind tracked some of the issues down to futures-rs unfortuantely:

The logs we have so far are below:

==1716606== Thread 25 magic_endpoint::
==1716606== Invalid read of size 8
==1716606==    at 0x38B867: core::sync::atomic::atomic_load (atomic.rs:3185)
==1716606==    by 0x38FBA7: core::sync::atomic::AtomicPtr<T>::load (atomic.rs:1325)
==1716606==    by 0xBAB2CB: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::dequeue (ready_to_run_queue.rs:60)
==1716606==    by 0xBA9FD7: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear (ready_to_run_queue.rs:102)
==1716606==    by 0xE9BA39: <futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut> as core::ops::drop::Drop>::drop (ready_to_run_queue.rs:119)
==1716606==    by 0xEFD092: core::ptr::drop_in_place<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0x94989D: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1749)
==1716606==    by 0xF42203: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xF02599: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xEF970E: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0xF1DAA6: core::ptr::drop_in_place<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::stream::map::Map<futures_util::stream::iter::Iter<alloc::vec::into_iter::IntoIter<alloc::sync::Arc<iroh_net::derp::map::DerpNode>>>,iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xF3FABE: core::ptr::drop_in_place<iroh_net::derp::http::client::Client::dial_region::{{closure}}> (client.rs:680)
==1716606==  Address 0x9eef4a8 is 1,352 bytes inside a block of size 1,368 free'd
==1716606==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0x10552AA: <alloc::alloc::Global as core::alloc::Allocator>::deallocate (alloc.rs:117)
==1716606==    by 0x105523F: <&A as core::alloc::Allocator>::deallocate (mod.rs:386)
==1716606==    by 0xF65894: <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop (sync.rs:3001)
==1716606==    by 0xF032A9: core::ptr::drop_in_place<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>,&alloc::alloc::Global>> (mod.rs:497)
==1716606==    by 0x94A158: drop<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>, &alloc::alloc::Global>> (mod.rs:987)
==1716606==    by 0x94A158: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1755)
==1716606==    by 0xF41943: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xEFCE39: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xC6987E: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::release_task (mod.rs:283)
==1716606==    by 0xC6AAA0: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear_head_all (mod.rs:573)
==1716606==    by 0xE9B7F9: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop (mod.rs:584)
==1716606==    by 0xEF9702: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==  Block was alloc'd at
==1716606==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0xB21C77: alloc (alloc.rs:98)
==1716606==    by 0xB21C77: alloc::alloc::Global::alloc_impl (alloc.rs:181)
==1716606==    by 0xB21A26: alloc::alloc::exchange_malloc (alloc.rs:241)
==1716606==    by 0x942BCE: new<alloc::sync::ArcInner<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>>> (boxed.rs:217)
==1716606==    by 0x942BCE: alloc::sync::Arc<T>::new (sync.rs:388)
==1716606==    by 0xC6EBB2: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::push (mod.rs:167)
==1716606==    by 0xAE5A1E: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next (buffer_unordered.rs:69)
==1716606==    by 0xAE5C58: futures_util::stream::stream::StreamExt::poll_next_unpin (mod.rs:1632)
==1716606==    by 0x4A2716: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll (next.rs:32)
==1716606==    by 0xE55E8D: iroh_net::derp::http::client::Client::dial_region::{{closure}} (client.rs:662)
==1716606==    by 0xE7380F: iroh_net::derp::http::client::Client::connect_0::{{closure}}::{{closure}} (client.rs:477)
==1716606==    by 0xC4FB2A: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll (instrument.rs:272)
==1716606==    by 0xE7120F: iroh_net::derp::http::client::Client::connect_0::{{closure}} (client.rs:463)
==1716606== 
==1716606== Invalid write of size 8
==1716606==    at 0x38E3A1: core::sync::atomic::atomic_store (atomic.rs:3170)
==1716606==    by 0x3902FA: core::sync::atomic::AtomicPtr<T>::store (atomic.rs:1355)
==1716606==    by 0xBAB677: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::enqueue (ready_to_run_queue.rs:41)
==1716606==    by 0xBAB36D: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::dequeue (ready_to_run_queue.rs:73)
==1716606==    by 0xBA9FD7: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear (ready_to_run_queue.rs:102)
==1716606==    by 0xE9BA39: <futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut> as core::ops::drop::Drop>::drop (ready_to_run_queue.rs:119)
==1716606==    by 0xEFD092: core::ptr::drop_in_place<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0x94989D: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1749)
==1716606==    by 0xF42203: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xF02599: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xEF970E: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0xF1DAA6: core::ptr::drop_in_place<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::stream::map::Map<futures_util::stream::iter::Iter<alloc::vec::into_iter::IntoIter<alloc::sync::Arc<iroh_net::derp::map::DerpNode>>>,iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==  Address 0x9eef4a8 is 1,352 bytes inside a block of size 1,368 free'd
==1716606==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0x10552AA: <alloc::alloc::Global as core::alloc::Allocator>::deallocate (alloc.rs:117)
==1716606==    by 0x105523F: <&A as core::alloc::Allocator>::deallocate (mod.rs:386)
==1716606==    by 0xF65894: <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop (sync.rs:3001)
==1716606==    by 0xF032A9: core::ptr::drop_in_place<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>,&alloc::alloc::Global>> (mod.rs:497)
==1716606==    by 0x94A158: drop<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>, &alloc::alloc::Global>> (mod.rs:987)
==1716606==    by 0x94A158: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1755)
==1716606==    by 0xF41943: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xEFCE39: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xC6987E: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::release_task (mod.rs:283)
==1716606==    by 0xC6AAA0: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear_head_all (mod.rs:573)
==1716606==    by 0xE9B7F9: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop (mod.rs:584)
==1716606==    by 0xEF9702: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==  Block was alloc'd at
==1716606==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0xB21C77: alloc (alloc.rs:98)
==1716606==    by 0xB21C77: alloc::alloc::Global::alloc_impl (alloc.rs:181)
==1716606==    by 0xB21A26: alloc::alloc::exchange_malloc (alloc.rs:241)
==1716606==    by 0x942BCE: new<alloc::sync::ArcInner<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>>> (boxed.rs:217)
==1716606==    by 0x942BCE: alloc::sync::Arc<T>::new (sync.rs:388)
==1716606==    by 0xC6EBB2: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::push (mod.rs:167)
==1716606==    by 0xAE5A1E: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next (buffer_unordered.rs:69)
==1716606==    by 0xAE5C58: futures_util::stream::stream::StreamExt::poll_next_unpin (mod.rs:1632)
==1716606==    by 0x4A2716: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll (next.rs:32)
==1716606==    by 0xE55E8D: iroh_net::derp::http::client::Client::dial_region::{{closure}} (client.rs:662)
==1716606==    by 0xE7380F: iroh_net::derp::http::client::Client::connect_0::{{closure}}::{{closure}} (client.rs:477)
==1716606==    by 0xC4FB2A: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll (instrument.rs:272)
==1716606==    by 0xE7120F: iroh_net::derp::http::client::Client::connect_0::{{closure}} (client.rs:463)
==1716606== 
==1716606== Invalid read of size 8
==1716606==    at 0x38B867: core::sync::atomic::atomic_load (atomic.rs:3185)
==1716606==    by 0x38FBA7: core::sync::atomic::AtomicPtr<T>::load (atomic.rs:1325)
==1716606==    by 0xBAB3B7: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::dequeue (ready_to_run_queue.rs:75)
==1716606==    by 0xBA9FD7: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear (ready_to_run_queue.rs:102)
==1716606==    by 0xE9BA39: <futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut> as core::ops::drop::Drop>::drop (ready_to_run_queue.rs:119)
==1716606==    by 0xEFD092: core::ptr::drop_in_place<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0x94989D: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1749)
==1716606==    by 0xF42203: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xF02599: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xEF970E: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0xF1DAA6: core::ptr::drop_in_place<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::stream::map::Map<futures_util::stream::iter::Iter<alloc::vec::into_iter::IntoIter<alloc::sync::Arc<iroh_net::derp::map::DerpNode>>>,iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xF3FABE: core::ptr::drop_in_place<iroh_net::derp::http::client::Client::dial_region::{{closure}}> (client.rs:680)
==1716606==  Address 0x9eef4a8 is 1,352 bytes inside a block of size 1,368 free'd
==1716606==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0x10552AA: <alloc::alloc::Global as core::alloc::Allocator>::deallocate (alloc.rs:117)
==1716606==    by 0x105523F: <&A as core::alloc::Allocator>::deallocate (mod.rs:386)
==1716606==    by 0xF65894: <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop (sync.rs:3001)
==1716606==    by 0xF032A9: core::ptr::drop_in_place<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>,&alloc::alloc::Global>> (mod.rs:497)
==1716606==    by 0x94A158: drop<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>, &alloc::alloc::Global>> (mod.rs:987)
==1716606==    by 0x94A158: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1755)
==1716606==    by 0xF41943: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xEFCE39: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xC6987E: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::release_task (mod.rs:283)
==1716606==    by 0xC6AAA0: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear_head_all (mod.rs:573)
==1716606==    by 0xE9B7F9: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop (mod.rs:584)
==1716606==    by 0xEF9702: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==  Block was alloc'd at
==1716606==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0xB21C77: alloc (alloc.rs:98)
==1716606==    by 0xB21C77: alloc::alloc::Global::alloc_impl (alloc.rs:181)
==1716606==    by 0xB21A26: alloc::alloc::exchange_malloc (alloc.rs:241)
==1716606==    by 0x942BCE: new<alloc::sync::ArcInner<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>>> (boxed.rs:217)
==1716606==    by 0x942BCE: alloc::sync::Arc<T>::new (sync.rs:388)
==1716606==    by 0xC6EBB2: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::push (mod.rs:167)
==1716606==    by 0xAE5A1E: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next (buffer_unordered.rs:69)
==1716606==    by 0xAE5C58: futures_util::stream::stream::StreamExt::poll_next_unpin (mod.rs:1632)
==1716606==    by 0x4A2716: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll (next.rs:32)
==1716606==    by 0xE55E8D: iroh_net::derp::http::client::Client::dial_region::{{closure}} (client.rs:662)
==1716606==    by 0xE7380F: iroh_net::derp::http::client::Client::connect_0::{{closure}}::{{closure}} (client.rs:477)
==1716606==    by 0xC4FB2A: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll (instrument.rs:272)
==1716606==    by 0xE7120F: iroh_net::derp::http::client::Client::connect_0::{{closure}} (client.rs:463)
==1716606== 
==1716606== Invalid read of size 8
==1716606==    at 0xF41928: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (atomic.rs:3236)
==1716606==    by 0xEFCE39: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xD6A51C: core::mem::drop (mod.rs:987)
==1716606==    by 0xBAA01A: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear (ready_to_run_queue.rs:105)
==1716606==    by 0xE9BA39: <futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut> as core::ops::drop::Drop>::drop (ready_to_run_queue.rs:119)
==1716606==    by 0xEFD092: core::ptr::drop_in_place<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0x94989D: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1749)
==1716606==    by 0xF42203: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xF02599: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xEF970E: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==    by 0xF1DAA6: core::ptr::drop_in_place<futures_util::stream::stream::buffer_unordered::BufferUnordered<futures_util::stream::stream::map::Map<futures_util::stream::iter::Iter<alloc::vec::into_iter::IntoIter<alloc::sync::Arc<iroh_net::derp::map::DerpNode>>>,iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xF3FABE: core::ptr::drop_in_place<iroh_net::derp::http::client::Client::dial_region::{{closure}}> (client.rs:680)
==1716606==  Address 0x9eeef60 is 0 bytes inside a block of size 1,368 free'd
==1716606==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0x10552AA: <alloc::alloc::Global as core::alloc::Allocator>::deallocate (alloc.rs:117)
==1716606==    by 0x105523F: <&A as core::alloc::Allocator>::deallocate (mod.rs:386)
==1716606==    by 0xF65894: <alloc::sync::Weak<T,A> as core::ops::drop::Drop>::drop (sync.rs:3001)
==1716606==    by 0xF032A9: core::ptr::drop_in_place<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>,&alloc::alloc::Global>> (mod.rs:497)
==1716606==    by 0x94A158: drop<alloc::sync::Weak<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>, &alloc::alloc::Global>> (mod.rs:987)
==1716606==    by 0x94A158: alloc::sync::Arc<T,A>::drop_slow (sync.rs:1755)
==1716606==    by 0xF41943: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop (sync.rs:2405)
==1716606==    by 0xEFCE39: core::ptr::drop_in_place<alloc::sync::Arc<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>>> (mod.rs:497)
==1716606==    by 0xC6987E: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::release_task (mod.rs:283)
==1716606==    by 0xC6AAA0: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear_head_all (mod.rs:573)
==1716606==    by 0xE9B7F9: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop (mod.rs:584)
==1716606==    by 0xEF9702: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<iroh_net::derp::http::client::Client::dial_region::{{closure}}::{{closure}}::{{closure}}>> (mod.rs:497)
==1716606==  Block was alloc'd at
==1716606==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1716606==    by 0xB21C77: alloc (alloc.rs:98)
==1716606==    by 0xB21C77: alloc::alloc::Global::alloc_impl (alloc.rs:181)
==1716606==    by 0xB21A26: alloc::alloc::exchange_malloc (alloc.rs:241)
==1716606==    by 0x942BCE: new<alloc::sync::ArcInner<futures_util::stream::futures_unordered::task::Task<iroh_net::derp::http::client::{impl#5}::dial_region::{async_fn#0}::{closure#0}::{async_block_env#0}>>> (boxed.rs:217)
==1716606==    by 0x942BCE: alloc::sync::Arc<T>::new (sync.rs:388)
==1716606==    by 0xC6EBB2: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::push (mod.rs:167)
==1716606==    by 0xAE5A1E: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next (buffer_unordered.rs:69)
==1716606==    by 0xAE5C58: futures_util::stream::stream::StreamExt::poll_next_unpin (mod.rs:1632)
==1716606==    by 0x4A2716: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll (next.rs:32)
==1716606==    by 0xE55E8D: iroh_net::derp::http::client::Client::dial_region::{{closure}} (client.rs:662)
==1716606==    by 0xE7380F: iroh_net::derp::http::client::Client::connect_0::{{closure}}::{{closure}} (client.rs:477)
==1716606==    by 0xC4FB2A: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll (instrument.rs:272)
==1716606==    by 0xE7120F: iroh_net::derp::http::client::Client::connect_0::{{closure}} (client.rs:463)
==1716606==

Reference: https://github.com/n0-computer/iroh/issues/1646

nikomatsakis commented 1 year ago

Some teams at Amazon have also encountered segfaults related to FuturesUnordered; they switched to Tokio's JoinSet. They've given me a backtrace, but it contains some proprietary symbol names and the like, and I'm not sure how useful it would be. I may anonymize it. It seems superficially similar to the backtraces I see here.

taiki-e commented 9 months ago

(https://github.com/rust-lang/futures-rs/pull/2809 may or may not have fixed this.)

nazar-pc commented 5 months ago

https://github.com/rust-lang/futures-rs/issues/2863 likely has (not very minimal, but still) reproduction of segfault that may have the same root cause

taiki-e commented 1 month ago

I think it is likely that 0.3.31, which fixed #2863, also fixed this problem.