search

bradyz / 2020_CARLA_challenge

"Learning by Cheating" (CoRL 2019) submission for the 2020 CARLA Challenge
181 stars 49 forks source link

Malloc size error in CARLA 9.10.1 #22

Closed aleallievi closed 3 years ago

aleallievi commented 3 years ago

Hi Brady,

I thought I got to the point of successfully implementing your agent in CARLA 9.10.1 and the newest version of scenario runner. Everything is working as expected, except for the simulator crashing every so often after running multiple consecutive routes (with no predictable pattern, as far as I can tell) and returning error:

terminating with uncaught exception of type clmdep_msgpack::v1::type_error: std::bad_cast
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=122688 LargeMemoryPoolOffset=253824 
Aborted (core dumped)

Is this something you have experienced before, or do you have an idea of what may be causing it?

Thanks in advance!

bradyz commented 3 years ago

I see this error every so often as well - I haven't yet pinpointed the cause but I suspect it has something to do with VRAM issues

aleallievi commented 3 years ago

Thanks for the quick feedback - yes, I think it might be a resource issue (although I am using the no-rendering-mode). I will do some more digging. I am not too well versed in the topic, but would appreciate recommendations (if you have any) on how one might go about reducing the model's inference memory footprint as a possible debugging step. Thanks!