search

cBournhonesque / lightyear

A networking library to make multiplayer games for the Bevy game engine
https://cbournhonesque.github.io/lightyear/book
Apache License 2.0
490 stars 49 forks source link

`entity.despawn()` causes panic/server crash #546

Closed SK83RJOSH closed 3 months ago

SK83RJOSH commented 3 months ago

Hello!

I have a case where an entity spawned on a client like so:

commands.spawn((
    OwnerId(local_client_id), // Where local_client_id = local peers ClientId
    Replicate {
        group: ReplicationGroup::new_id(local_client_id.to_bits()),
        ..default()
    },
));

And configured on the server like so:

fn replicate_owned_entities(
    mut commands: Commands,
    mut owned_entities_reader: EventReader<ComponentInsertEvent<OwnerId>>,
) {
    for event in owned_entities_reader.read() {
        info!("received owned entity: {:?}", event);
        let client_id = event.context();
        let entity = event.entity();

        if let Some(mut e) = commands.get_entity(entity) {
            e.insert(Replicate {
                target: ReplicationTarget {
                    target: NetworkTarget::AllExceptSingle(*client_id),
                },
                sync: SyncTarget {
                    prediction: NetworkTarget::None,
                    interpolation: NetworkTarget::AllExceptSingle(*client_id),
                },
                controlled_by: ControlledBy {
                    target: NetworkTarget::Single(*client_id),
                    ..default()
                },
                group: ReplicationGroup::new_id(client_id.to_bits()),
                ..default()
            });
        }
    }
}

And despawned on the owning client side:

commands.entity(entity_id).despawn(); // for any locally "owned" object

Results in the server panicking:

thread 'main' panicked at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17:
Resource requested by lightyear::server::clients::systems::handle_controlled_by_remove does not exist: lightyear::server::connection::ConnectionManager
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `lightyear::server::networking::receive`!
Full Backtrace ```rust thread 'main' panicked at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::server::clients::systems::handle_controlled_by_remove does not exist: lightyear::server::connection::ConnectionManager stack backtrace: 0: std::panicking::begin_panic_handler at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077/library\std\src\panicking.rs:652 1: core::panicking::panic_fmt at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077/library\core\src\panicking.rs:72 2: bevy_ecs::system::system_param::impl$4::get_param::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534 3: enum2$,bevy_ecs::component::TickCells> > >::unwrap_or_else,bevy_ecs::component::TickCells>,bevy_ecs::system::system_param::impl$4::get_param::closure_env$ at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\option.rs:977 4: bevy_ecs::system::system_param::impl$4::get_param at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:531 5: bevy_ecs::system::system_param::impl$85::get_param,tuple$<> >,bevy_ecs::system::query::Query,tuple$<> >,bevy_e at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:1474 6: bevy_ecs::system::function_system::impl$7::run_unsafe,bevy_ecs::system::query::Query at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\observer\runner.rs:409 8: bevy_ecs::observer::impl$2::invoke::closure$0,core::iter::adapters::cloned::Cloned > > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\observer\mod.rs:192 9: core::ops::function::impls::impl$3::call_mut,ref$)> > >,bevy_ecs::observer::impl$2 at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\ops\function.rs:294 10: core::iter::traits::iterator::Iterator::for_each::call::closure$0,ref$)> >,ref_mut$),tuple$<>,core::iter::traits::iterator::Iterator::for_each::c at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\hashbrown-0.14.5\src\map.rs:4749 12: hashbrown::raw::inner::RawIterRange)> >::fold_impl)>,tuple$<>,hashbrown::map::impl$63::fold::closure_env$0),tuple$<>,core::iter::traits::iterator::Iterator::for_each::call::closur at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\hashbrown-0.14.5\src\map.rs:4747 15: core::iter::traits::iterator::Iterator::for_each)>,ref_mut$,core::iter::adapters::cloned::Cloned > > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\observer\mod.rs:216 17: core::iter::traits::iterator::Iterator::for_each::call::closure$0,core::iter::adapters::cloned::Cloned > at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\iter\traits\iterator.rs:815 18: core::iter::adapters::map::map_fold::closure$0,bevy_ecs::component::ComponentId,tuple$<>,bevy_ecs::component::ComponentId (*)(ref$),core::iter::traits::iterator::Iterator::for_each:: at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\iter\adapters\map.rs:89 19: core::slice::iter::impl$181::fold,core::iter::adapters::map::map_fold::closure_env$0,bevy_ecs::component::ComponentId,tuple$<>,bevy_ecs::component::ComponentId (*)(ref$,bevy_ecs::component::ComponentId,tuple$<>,core::iter::traits::iterator::Iterator::for_each::call::closure_env$0 >,bevy_ecs::observer::impl$2::invoke::closure_env$1,core::iter::adapters::cloned::Cloned,core::iter::adapters::cloned::Cloned > > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\observer\mod.rs:214 24: bevy_ecs::world::deferred_world::DeferredWorld::trigger_observers > > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\deferred_world.rs:353 25: bevy_ecs::world::entity_ref::EntityWorldMut::despawn at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\entity_ref.rs:1227 26: bevy_ecs::world::World::despawn at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1105 27: bevy_hierarchy::hierarchy::despawn_with_children_recursive_inner at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_hierarchy-0.14.0\src\hierarchy.rs:44 28: bevy_hierarchy::hierarchy::despawn_with_children_recursive at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_hierarchy-0.14.0\src\hierarchy.rs:33 29: bevy_hierarchy::hierarchy::impl$3::despawn_recursive at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_hierarchy-0.14.0\src\hierarchy.rs:120 30: lightyear::shared::replication::receive::GroupChannel::apply_actions_message at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\shared\replication\receive.rs:855 31: lightyear::shared::replication::receive::impl$1::apply_world::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\shared\replication\receive.rs:524 32: core::iter::traits::iterator::Iterator::for_each::call::closure$0,ref_mut$ >,lightyear::shared::replication::receive::impl$1 at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\iter\traits\iterator.rs:815 33: hashbrown::map::impl$66::fold::closure$0,core::iter::traits::iterator::Iterator::for_each::call::closure_env$0 >::fold_impl,tuple$<>,hashbrown::map::impl$66::fold::closure_env$0,core::iter::traits::iterator::Iterator::for_each::call::closure_env$0,lightyear::shared::replication::receive::impl$1::apply_world::cl at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\iter\traits\iterator.rs:818 38: lightyear::shared::replication::receive::ReplicationReceiver::apply_world at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\shared\replication\receive.rs:496 39: lightyear::server::connection::Connection::receive at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\connection.rs:667 40: lightyear::server::connection::impl$1::receive::closure$0::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\connection.rs:342 41: bevy_ecs::world::World::resource_scope,enum2$ > >,lightyear::server::connection::impl$1::receive::closure$0::closure_env$0> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1886 42: lightyear::server::connection::impl$1::receive::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\connection.rs:340 43: core::iter::traits::iterator::Iterator::try_for_each::call::closure$0 >,ref_mut$ >,enum2$,enum2$,lightyear::server::connection::Connection>,tuple$<>,core::iter::traits::iterator::Iterator::try_for_each::call::closure_env$0,lightyear::server::connection::Connection>,lightyear::server::connection::impl$1::receive::closure_env$0,enum2$,lightyear::server::networking::receive::closure$0::closure$0::closure$0::closure_env$0> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1886 49: lightyear::server::networking::receive::closure$0::closure$0::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\networking.rs:80 50: bevy_ecs::world::World::resource_scope,lightyear::server::networking::receive::closure$0::closure$0::closure_env$0> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1886 51: lightyear::server::networking::receive::closure$0::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\networking.rs:78 52: bevy_ecs::world::World::resource_scope,lightyear::server::networking::receive::closure$0::closure_env$0> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1886 53: lightyear::server::networking::receive::closure$0 at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\networking.rs:76 54: bevy_ecs::world::World::resource_scope,lightyear::server::networking::receive::closure_env$0> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:1886 55: lightyear::server::networking::receive at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lightyear-0.16.3\src\server\networking.rs:75 56: core::ops::function::FnMut::call_mut),tuple$ > > at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\ops\function.rs:166 57: core::ops::function::impls::impl$3::call_mut >,void (*)(ref_mut$)> at /rustc/aed2187d53b8789e3a37f50ae36f894a2a679077\library\core\src\ops\function.rs:294 58: bevy_ecs::system::exclusive_function_system::impl$2::run::call_inner,ref_mut$)> > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\exclusive_function_system.rs:218 59: bevy_ecs::system::exclusive_function_system::impl$2::run,void (*)(ref_mut$)> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\exclusive_function_system.rs:221 60: bevy_ecs::system::exclusive_function_system::impl$1::run::closure$0)> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\exclusive_function_system.rs:111 61: bevy_ecs::world::World::last_change_tick_scope,bevy_ecs::system::exclusive_function_system::impl$1::run::closure_env$0)> > at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\world\mod.rs:2215 62: bevy_ecs::system::exclusive_function_system::impl$1::run)> at C:\Users\Josh\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\exclusive_function_system.rs:103 ```

The error hints at ConnectionManager being the issue, but after a lot of trying to grok the associated code, I wasn't able to determine if we're missing something in our server plugin setup, or if we've uncovered some nasty bug. So I wanted to create an issue and see if you might have any leads. Do note that everything up until the despawn works just fine, which I assume means the ConnectionManager indeed exists before the panic, but I haven't gotten around to dumping the ECS world to confirm. 🙂

cBournhonesque commented 3 months ago

Are the client and server running in separate processes?

Yes i've seen that error; for some reason when my observers trigger (in this case this one https://github.com/cBournhonesque/lightyear/blob/main/lightyear/src/server/clients.rs#L73), I have errors telling me that a resource like ConnectionManager doesn't exist. This doesn't make sense to me since ConnectionManager should exist at all times!

As a quick fix I can update the argument to be Option<ConnectionManager>

cBournhonesque commented 3 months ago

Does this fix it? https://github.com/cBournhonesque/lightyear/pull/549

cBournhonesque commented 3 months ago

Ah I know, it's because I trigger the observer while inside a world.resource_scope so the resource has been removed

cBournhonesque commented 3 months ago

This should be fixed from https://github.com/cBournhonesque/lightyear/pull/549

SK83RJOSH commented 3 months ago

Heya, just wanted to follow up and say it seems to still be occurring on my end when using the latest commit:

Full Backtrace ```rust thread 'main' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::server::clients::systems::handle_controlled_by_remove does not exist: lightyear::server::connection::ConnectionManager note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Encountered a panic in system `lightyear::server::networking::receive`! thread 'Compute Task Pool (4)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (6)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (4)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (6)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (4)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (6)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (4)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'thread 'Compute Task Pool (4)Compute Task Pool (6)' panicked at ' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rsC:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs::534534::1717: : Resource requested by lightyear::shared::replication::plugin::send::ReplicationSendPlugin::tick_send_interval_timer does not exist: lightyear::shared::time_manager::TimeManagerResource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::replication::plugin::send::ReplicationSendPlugin::tick_send_interval_timer`! Encountered a panic in system `lightyear::shared::events::systems::push_component_eventsthread '`! Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (6)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (5)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! thread 'Compute Task Pool (7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17: Resource requested by lightyear::shared::events::systems::push_component_events does not exist: lightyear::protocol::component::ComponentRegistry Encountered a panic in system `lightyear::shared::events::systems::push_component_events`! em `bevy_app::main_schedule7)' panicked at C:\Users\Redacted\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_ecs-0.14.0\src\system\system_param.rs:534:17:::Main::run_main`! ```

This one is a bit hard to make a minimal repro for and our time zones are a bit out of sync, but I'll see what I can do tomorrow. Sorry in the meantime though, this one feels bound to be a bit frustrating... 😓

cBournhonesque commented 3 months ago

Ah I see, that's interesting; I opened an issue here https://github.com/cBournhonesque/lightyear/issues/550 I think I know how to fix it. I have to avoid doing archetype changes while inside world.resource_scope

cBournhonesque commented 3 months ago

@SK83RJOSH I finally got around to fixing it! I also added a unit test that exactly reproduces your situation, so it should be good now :)

SK83RJOSH commented 3 months ago

I can confirm it's fixed on our end! Thank you so much ❤️