Traffic Manager leaks memory OR std::out_of_range

[yasser-h-khalil]

Hello,

I am using Traffic Manager to spawn surrounding vehicles (autopilot mode), I am also spawning an ego vehicle along with some sensors. I am continuously resetting the environment as I am training Carla on Reinforcement Learning. I noticed that Traffic Manager continuously leaks my GPU memory and if left for long hours it results in out of memory:

`` terminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_castterminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_cast terminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_cast

Signal 6 caught.
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
Signal 6 caught.
CommonUnixCrashHandler: Signal=6
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=123824 LargeMemoryPoolOffset=254960 
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.
Aborted (core dumped)

How do we solve this issue? Are we supposed to turn the synchronization on the traffic manager off when we turn off the synchronization on the world (just for some time) to spawn the ego vehicle and sensors while resetting?

Sometimes even, if the memory is still not flooded, I get this error:

terminate called after throwing an instance of 'std::out_of_range'
  what():  _Map_base::at
Fatal Python error: Aborted

I made sure that all my surrounding vehicles, ego, and sensors are destroyed before resetting, so I am not sure why I am getting this error after many hours of execution.

I am using the latest version of Carla (0.9.10).

Any help is much appreciated! Thanks all and especially Carla team for their great efforts.

[yasser-h-khalil]

I would like to document what I figured out so far.

1) Set the synchronous mode of traffic manager only once at the beginning. Don't turn off and on when resetting. 2) When turning off the synchronous mode of the client don't set the fixed_delta_seconds to None. 3) I found that Town05 consumes more memory than Town03

I still get the following error when left for execution for long periods of time.

terminate called after throwing an instance of 'std::out_of_range' what(): _Map_base::at Fatal Python error: Aborted

I still can't figure out what is the problem behind it. Any help is appreciated.

[totalmayhem96]

Any ideas on this? Running a batch of more than 20 route scenarios and I run out of GPU memory. Seeing as I need to run batches of probably 100+, this could be a problem.

[corkyw10]

@jackbart94 Would you be able to follow up on this please?

[jackbart94]

Could anyone provide a code snippet to reproduce this? Please, highlight also how you are creating Traffic Manager and how many times do you use get_trafficmanager().

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.