search

eclipse-sumo / sumo

Eclipse SUMO is an open source, highly portable, microscopic and continuous traffic simulation package designed to handle large networks. It allows for intermodal simulation including pedestrians and comes with a large set of tools for scenario creation.
https://eclipse.dev/sumo
Eclipse Public License 2.0
2.37k stars 1.37k forks source link

Crashes when Co-simulated with Carla #13322

Closed cmpute closed 7 months ago

cmpute commented 1 year ago

In short, I met various crashes and segfaults when I co-simulated SUMO with Carla. I have no idea what the possible reasons behind these errors and any suggestions on is how to troubleshoot this is appreciated!

System info:

Unfortunately, I can't provide the code to reproduce this and it's hard to get a minimal reproducing example. I'm using the Traci / libsumo API with the Carla Python API to write the cosimulation code, however the simulation always crashes after a few simulation steps with one of my co-simulation settings.

If I use libsumo, the simulation always ends with segfaults. Below are an example of the segfault stack trace from gdb:

Libsumo Segfault Stacktrace ```log #0 0x00007fffe03a7382 in std::string toString(std::string const&, long) () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #1 0x00007fffe04b0ef7 in void PlainXMLFormatter::writeAttr(std::ostream&, SumoXMLAttr, std::string const&) () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #2 0x00007fffe0639b31 in MSFCDExport::write(OutputDevice&, long long, bool) () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #3 0x00007fffe07f6380 in MSNet::writeOutput() () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #4 0x00007fffe07f7c38 in MSNet::simulationStep() () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #5 0x00007fffe039b1d3 in libsumo::Simulation::step(double) () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #6 0x00007fffe01b20ff in _wrap_simulation_step () from /home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/libsumo/_libsumo.so #7 0x00000000004aef97 in _PyMethodDef_RawFastCallKeywords (method=, self=0x7fffe161f590, args=0x7fffd260ea40, nargs=, kwnames=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:693 #8 0x00000000004ae9f0 in _PyCFunction_FastCallKeywords (kwnames=, nargs=, args=0x7fffd260ea40, func=0x7fffd7292cd0) at /usr/local/src/conda/python-3.7.16/Objects/call.c:732 #9 call_function (pp_stack=0x7fffffffc8e0, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4568 #10 0x00000000004ab36a in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3093 #11 0x00000000004a5a81 in PyEval_EvalFrameEx (throwflag=, f=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:547 #12 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=, kwnames=, kwargs=0x7fffd26103b0, kwcount=, kwstep=1, defs=0x7fffe16431a8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff77750f0, qualname=0x7fffe1635c70) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3930 #13 0x00000000004b9eac in _PyFunction_FastCallKeywords (func=, stack=0x7fffd26103a8, nargs=1, kwnames=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:433 #14 0x00000000004ae8df in call_function (pp_stack=0x7fffffffcbb8, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4616 #15 0x00000000004a7651 in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3124 #16 0x000000000054b040 in PyEval_EvalFrameEx (throwflag=, f=0x7fffd2610220) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:544 #17 function_code_fastcall (co=, args=, nargs=, globals=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:283 #18 0x00000000004ae8df in call_function (pp_stack=0x7fffffffcd70, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4616 #19 0x00000000004ab36a in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3093 #20 0x00000000004c0d96 in PyEval_EvalFrameEx (throwflag=, f=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:544 #21 function_code_fastcall (globals=, nargs=, --Type for more, q to quit, c to continue without paging-- nargs@entry=, args=, co=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:283 #22 _PyFunction_FastCallDict (func=, args=0x7fffffffced0, nargs=, kwargs=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:322 #23 0x00000000004c9a80 in _PyObject_FastCallDict (kwargs=0x7fffd25fc9b0, nargs=3, args=, callable=0x7fffd31a0b90) at /usr/local/src/conda/python-3.7.16/Objects/call.c:84 #24 _PyObject_Call_Prepend (kwargs=0x7fffd25fc9b0, kwargs@entry=, args=, args@entry=, obj=0x7fffd3398a10, obj@entry=, callable=0x7fffd31a0b90, callable@entry=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:906 #25 method_call (method=method@entry=0x7ffff71bd0a0, args=, kwargs=0x7fffd25fc9b0) at /usr/local/src/conda/python-3.7.16/Objects/classobject.c:309 #26 0x00000000004c7170 in PyObject_Call (callable=0x7ffff71bd0a0, args=, kwargs=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:245 #27 0x00000000004a8b84 in do_call_core (kwdict=0x7fffd25fc9b0, callargs=0x7fffd315cdc0, func=0x7ffff71bd0a0) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4645 #28 _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3191 #29 0x00000000004a5a81 in PyEval_EvalFrameEx (throwflag=, f=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:547 #30 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=, kwnames=, kwargs=0x0, kwcount=, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff775f1b0, qualname=0x7fffd319edf0) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3930 #31 0x00000000004c0f57 in _PyFunction_FastCallDict (func=, args=0x7fffffffd240, nargs=3, kwargs=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:367 #32 0x00000000004c6abe in _PyObject_Call_Prepend (callable=0x7fffd31a07a0, obj=, args=0x7fffd31cb320, kwargs=0x0) at /usr/local/src/conda/python-3.7.16/Objects/call.c:906 #33 0x0000000000578b67 in slot_tp_call (self=0x7fffd2611530, args=0x7fffd31cb320, kwds=0x0) at /usr/local/src/conda/python-3.7.16/Objects/typeobject.c:6402 #34 0x00000000004b77a0 in _PyObject_FastCallKeywords (callable=0x7fffd2611530, stack=, nargs=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/errors.c:176 #35 0x00000000004aea19 in call_function (pp_stack=0x7fffffffd430, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4619 #36 0x00000000004ab36a in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3093 #37 0x00000000004b9d16 in PyEval_EvalFrameEx (throwflag=, f=0x7fffd2602050) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:544 #38 function_code_fastcall (globals=, nargs=1, nargs@entry=, args=, co=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:283 #39 _PyFunction_FastCallKeywords (func=, stack=0x7fffd26001c8, nargs=1, kwnames=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:408 #40 0x00000000004ae8df in call_function (pp_stack=0x7fffffffd600, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4616 #41 0x00000000004a777e in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3110 --Type for more, q to quit, c to continue without paging-- #42 0x00000000004b9d16 in PyEval_EvalFrameEx (throwflag=, f=0x7fffd2600050) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:544 #43 function_code_fastcall (globals=, nargs=1, nargs@entry=, args=, co=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:283 #44 _PyFunction_FastCallKeywords (func=, stack=0x7ffff76ad5c0, nargs=1, kwnames=) at /usr/local/src/conda/python-3.7.16/Objects/call.c:408 #45 0x00000000004ae8df in call_function (pp_stack=0x7fffffffd7d0, oparg=, kwnames=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:4616 #46 0x00000000004a777e in _PyEval_EvalFrameDefault (f=, throwflag=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3110 #47 0x00000000004a5a81 in PyEval_EvalFrameEx (throwflag=, f=) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:547 #48 _PyEval_EvalCodeWithName (_co=, globals=, locals=, args=, argcount=, kwnames=, kwargs=0x0, kwcount=, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3930 #49 0x00000000004a5879 in PyEval_EvalCodeEx (_co=, globals=, locals=, args=, argcount=, kws=, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:3959 #50 0x000000000054a8db in PyEval_EvalCode (co=co@entry=0x7ffff7635030, globals=globals@entry=0x7ffff7717e60, locals=locals@entry=0x7ffff7717e60) at /usr/local/src/conda/python-3.7.16/Python/ceval.c:524 #51 0x0000000000564e03 in run_mod (mod=, filename=, globals=0x7ffff7717e60, locals=0x7ffff7717e60, flags=, arena=) at /usr/local/src/conda/python-3.7.16/Python/pythonrun.c:1037 #52 0x000000000056b857 in PyRun_FileExFlags (fp=0x710310, filename_str=, start=, globals=0x7ffff7717e60, locals=0x7ffff7717e60, closeit=1, flags=0x7fffffffdac0) at /usr/local/src/conda/python-3.7.16/Python/pythonrun.c:990 #53 0x000000000056ae34 in PyRun_SimpleFileExFlags (fp=0x710310, filename=0x7ffff76d2dc0 "examples/example_carla_with_av.py", closeit=1, flags=0x7fffffffdac0) at /usr/local/src/conda/python-3.7.16/Python/pythonrun.c:429 #54 0x000000000053fea3 in pymain_run_file (p_cf=0x7fffffffdac0, filename=, fp=0x710310) at /usr/local/src/conda/python-3.7.16/Modules/main.c:470 #55 pymain_run_filename (cf=0x7fffffffdac0, pymain=0x7fffffffdbd0) at /usr/local/src/conda/python-3.7.16/Modules/main.c:1660 #56 pymain_run_python (pymain=0x7fffffffdbd0) at /usr/local/src/conda/python-3.7.16/Modules/main.c:2970 #57 pymain_main (pymain=0x7fffffffdbd0) at /usr/local/src/conda/python-3.7.16/Modules/main.c:3517 #58 0x000000000053fb3c in _Py_UnixMain (argc=, argv=) at /usr/local/src/conda/python-3.7.16/Modules/main.c:3552 #59 0x00007ffff7c50083 in __libc_start_main (main=0x53faf0
, argc=2, argv=0x7fffffffdd38, init=, fini=, rtld_fini=, stack_end=0x7fffffffdd28) at ../csu/libc-start.c:308 #60 0x000000000053f9ee in _start () at /usr/local/src/conda/python-3.7.16/Modules/itertoolsmodule.c:2515 ```

If I use traci, the simulation will ends with various errors, stacktrace examples are listed below:

Traci Error Stacktrace 1 ```log Traceback (most recent call last): File "/home/jacobz/MTL-Simulation-Platform/mtlsp/physics/carla/sensors.py", line 97, in store_data sm.push_variable(self._agent.id, self._name, data) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/state_manager.py", line 69, in push_variable Traceback (most recent call last): File "examples/example_carla_with_av.py", line 84, in current_time = traci.simulation.getTime() sim.run() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 190, in run File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/_simulation.py", line 257, in getTime self.step() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 153, in step self.step_pipeline(self, self.ctx) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/pipeline.py", line 16, in __call__ output = element.executable(*args, **kwargs) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/physics/carla/plugin.py", line 197, in step return self._getUniversal(tc.VAR_TIME) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 147, in _getUniversal sumo_actors[agent_id] = BridgeHelper.get_sumo_actor(agent_id) return _parse(self._retValFunc, varID, self._getCmd(varID, objectID, format, *values)) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 152, in _getCmd File "/home/jacobz/MTL-Simulation-Platform/mtlsp/physics/carla/helper.py", line 188, in get_sumo_actor r = self._connection._sendCmd(self._cmdGetID, varID, objID, format, *values) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 225, in _sendCmd vclass = SumoActorClass(traci.vehicle.getVehicleClass(sumo_id)) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/_vehicletype.py", line 131, in getVehicleClass return self._getUniversal(tc.VAR_VEHICLECLASS, typeID) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 147, in _getUniversal return self._sendExact() File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 142, in _sendExact return _parse(self._retValFunc, varID, self._getCmd(varID, objectID, format, *values)) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 158, in _getCmd raise TraCIException(err, prefix[1], _RESULTS[prefix[2]]) % (response, retVarID, objectID, self._cmdGetID, varID, objID)) KeyError: 73 traci.exceptions.FatalTraCIError: Received answer 164,0, for command 164,73,0. Error: tcpip::Socket::recvAndCheck @ recv: peer shutdownvehicles TOT 1 ACT 1 BUF 0) Quitting (on error). ```
Traci Error Stacktrace 2 ```log Traceback (most recent call last): File "examples/example_carla_with_av.py", line 84, in sim.run() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 190, in run self.step() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 153, in step self.step_pipeline(self, self.ctx) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/pipeline.py", line 16, in __call__ output = element.executable(*args, **kwargs) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 166, in sumo_step traci.simulationStep() File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/main.py", line 199, in simulationStep return connection.check().simulationStep(step) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 363, in simulationStep result = self._sendCmd(tc.CMD_SIMSTEP, None, None, "D", step) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 225, in _sendCmd return self._sendExact() File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 135, in _sendExact raise FatalTraCIError("connection closed by SUMO") traci.exceptions.FatalTraCIError: connection closed by SUMO ```
Traci Error Stacktrace 3 ```log Traceback (most recent call last): File "examples/example_carla_with_av.py", line 84, in sim.run() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 190, in run self.step() File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 153, in step self.step_pipeline(self, self.ctx) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/pipeline.py", line 16, in __call__ output = element.executable(*args, **kwargs) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/physics/carla/plugin.py", line 197, in step sumo_actors[agent_id] = BridgeHelper.get_sumo_actor(agent_id) File "/home/jacobz/MTL-Simulation-Platform/mtlsp/physics/carla/helper.py", line 191, in get_sumo_actor length = traci.vehicle.getLength(sumo_id) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/_vehicletype.py", line 43, in getLength return self._getUniversal(tc.VAR_LENGTH, typeID) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 147, in _getUniversal return _parse(self._retValFunc, varID, self._getCmd(varID, objectID, format, *values)) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/domain.py", line 152, in _getCmd r = self._connection._sendCmd(self._cmdGetID, varID, objID, format, *values) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 225, in _sendCmd return self._sendExact() File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 138, in _sendExact err = result.readString() File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/storage.py", line 69, in readString return str(self.read("!%ss" % length)[0].decode("latin1")) File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/storage.py", line 38, in read return struct.unpack(format, self._content[oldPos:self._pos]) struct.error: unpack requires a buffer of 297026560 bytes Error: tcpip::Socket::recvAndCheck @ recv: peer shutdown1 ACT 1 BUF 0) Quitting (on error). ```

Note that all these errors happened with the same script and same co-simulation settings. Another thing I found hard to understand was that the script won't crash if I don't add vehicles to SUMO and don't get sensor data from the Carla side (although I'm not sure which one introduces the crash).

Again, any help would be appreciated, thanks in advance!

namdre commented 1 year ago

You could try to get a reproducible failing test that runs without Carla by logging all traci commands (https://sumo.dlr.de/docs/TraCI/Interfacing_TraCI_from_Python.html#generating_a_log_of_all_traci_commands) and then attach this test case.

cmpute commented 1 year ago

Thanks for the suggestion! I tried to add the trace logging, but the logging isn't always successful (sometimes Traci will throw an error before anything is written to the trace file).

Here is an example with successful trace output:

Trace Logs ``` traci.start(['sumo', '-c', 'examples/maps/CarlaTown04/Town04.sumocfg', '--step-length', '0.1', '--fcd-output', 'examples/output/0/fcd_all.xml', '--fcd-output.acceleration'], port=None, label='default') traci.simulation.getTime() traci.vehicle.add('hero', 'r_1', typeID='DEFAULT_VEHTYPE', depart=None, departLane='first', departPos=0, departSpeed=0, arrivalLane='current', arrivalPos='max', arrivalSpeed='current', fromTaz='', toTaz='', line='', personCapacity=0, personNumber=0) traci.simulation.getTime() traci.vehicle.getTypeID('hero') traci.vehicle.getVehicleClass('hero') traci.vehicle.getColor('hero') traci.vehicle.getLength('hero') traci.vehicle.getWidth('hero') traci.vehicle.getHeight('hero') traci.vehicle.getPosition3D('hero') traci.vehicle.getSlope('hero') traci.vehicle.getAngle('hero') traci.vehicle.getSignals('hero') traci.simulation.getTime() traci.vehicle.setSpeedMode('hero', 31) traci.vehicle.setLaneChangeMode('hero', 1621) traci.vehicle.setColor('hero', (255, 255, 0)) traci.vehicle.setColor('hero', (255, 255, 0)) traci.vehicle.getIDList() traci.vehicle.remove('hero') traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getTime() traci.vehicle.setSpeedMode('0', 31) traci.vehicle.setLaneChangeMode('0', 1621) traci.vehicle.setColor('0', (255, 255, 0)) traci.vehicle.setColor('0', (255, 255, 0)) traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getTime() traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getTime() traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getTime() traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() traci.simulation.getTime() traci.vehicle.getIDList() traci.vehicle.getTypeID('0') traci.vehicle.getVehicleClass('0') traci.vehicle.getColor('0') traci.vehicle.getLength('0') traci.vehicle.getWidth('0') traci.vehicle.getHeight('0') traci.vehicle.getPosition3D('0') traci.vehicle.getSlope('0') traci.vehicle.getAngle('0') traci.vehicle.getSignals('0') traci.simulation.getTime() traci.simulation.getMinExpectedNumber() traci.vehicle.getIDList() traci.vehicle.add('added_0', 'r_0', typeID='DEFAULT_VEHTYPE', depart=None, departLane='first', departPos=0, departSpeed=0, arrivalLane='current', arrivalPos='max', arrivalSpeed='current', fromTaz='', toTaz='', line='', personCapacity=0, personNumber=0) traci.simulation.getTime() traci.vehicle.getTypeID('added_0') traci.vehicle.getVehicleClass('added_0') traci.vehicle.getColor('added_0') traci.vehicle.getLength('added_0') traci.vehicle.getWidth('added_0') traci.vehicle.getHeight('added_0') traci.vehicle.getPosition3D('added_0') traci.vehicle.getSlope('added_0') traci.vehicle.getAngle('added_0') traci.vehicle.getSignals('added_0') traci.vehicle.setSpeedMode('added_0', 31) traci.vehicle.setLaneChangeMode('added_0', 1621) traci.vehicle.setColor('added_0', (255, 255, 0)) traci.vehicle.setColor('added_0', (255, 255, 0)) traci.simulation.getTime() traci.simulation.getTime() traci.simulation.getTime() traci.vehicle.moveToXY('0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.vehicle.moveToXY('added_0', edgeID='', lane=-1, x=100, y=46, angle=0.2, keepRoute=2) traci.simulation.getMinExpectedNumber() traci.simulationStep() ```

And here's the Python stack trace

Traceback (most recent call last):
  File "examples/example_carla_with_av.py", line 84, in <module>
    sim.run()
  File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 190, in run
    self.step()
  File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 153, in step
    self.step_pipeline(self, self.ctx)
  File "/home/jacobz/MTL-Simulation-Platform/mtlsp/pipeline.py", line 16, in __call__
    output = element.executable(*args, **kwargs)
  File "/home/jacobz/MTL-Simulation-Platform/mtlsp/simulator.py", line 166, in sumo_step
    traci.simulationStep()
  File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/main.py", line 199, in simulationStep
    return connection.check().simulationStep(step)
  File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 363, in simulationStep
    result = self._sendCmd(tc.CMD_SIMSTEP, None, None, "D", step)
  File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 225, in _sendCmd
    return self._sendExact()
  File "/home/jacobz/miniconda3/envs/carla/lib/python3.7/site-packages/traci/connection.py", line 135, in _sendExact
    raise FatalTraCIError("connection closed by SUMO")
traci.exceptions.FatalTraCIError: connection closed by SUMO

I tried the trace commands locally and it does seem to be reproducing the behavior. But I don't know what caused the issue..

namdre commented 1 year ago

sorry, for the ambiguous wording: Having a trace file is a necessary but insufficient condition for replicating the problem. I also need the sumocfg and all files referenced therein (preferably as an attacked zip archive).

behrisch commented 7 months ago

I tried to run your example with the Town04 scenario of the current CARLA Release 0.9.15 and it fails because the route "r_1" you used for your vehicle "hero" does not exist. Since the scenario did not seem to change from 0.9.14, the problem might be in you code. Please add more information in another comment if you fixed this and problems persist. Although I have to warn you that it is hard for us to reproduce CARLA problems because of https://github.com/carla-simulator/carla/issues/6484

behrisch commented 7 months ago

I could also not reproduce the libsumo segfault.