search

lgsvl / simulator

A ROS/ROS2 Multi-robot Simulator for Autonomous Vehicles
Other
2.29k stars 779 forks source link

svl simulator shut down every 20 seconds #1396

Open Joonyi0130 opened 3 years ago

Joonyi0130 commented 3 years ago

when i start Simulations with 'Visual scenario editor' Template, it just play only 20 seconds. everytime when i change scenarios, which is make with visual editor, it happened. if i choose Random Traffic or Python API, it goes well after 20 sec any idea what is wrong with my Simulation setting? Do i miss something?

EricBoiseLGSVL commented 3 years ago

Do you have a player.log we can look at? What OS, AD stack, GPU are you running? @KarByc ?

KarByc commented 3 years ago

@Joonyi0130 Is there any NPC with NPCWaypointBehaviour added to the scenario?

Joonyi0130 commented 3 years ago

@EricBoiseLGSVL i'm using ubuntu 20.04.1 and GTX1660 supper. where can i get player.log? can't get with grep HAHA nothing searched

Joonyi0130 commented 3 years ago

@KarByc yes i made both NPCAgent and PedestrianAgent. in Visual Editor, it plays longer than 20 seconds. it only happens in Simulations

Joonyi0130 commented 3 years ago

when reached 20 sec, there's an message on info tab in simulation and it shows like 0.2sec lol so i recorded it, hope it helps to figure out what happens Screenshot_2021-04-28_09-12-35

KarByc commented 3 years ago

@Joonyi0130 Thanks for the feedback. Looks like the Player.log may help us with finding out the issue. Can you attach a Player.log file after a VSE scenario run?

GabrielZZZ commented 3 years ago

I encountered the same issue. @KarByc Here is my Player.log:

`[CONN] HTTP 200 OK WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed WARNING: Shader Did you use #pragma only_renderers and omit this platform? WARNING: Shader Unsupported: 'Shader Graphs/VehicleLightsShader' - All passes removed WARNING: Shader Did you use #pragma only_renderers and omit this platform? found manifest at 1e531ff0-8ce5-4d26-9dda-bbd7a4e2c930/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5185620.so found manifest at 328b2afb-78c4-4c86-8a18-62015194a3fe/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bb61d0.so found manifest at 7affd0d4-e9a5-4100-8a0f-e914c1176551/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc3480.so found manifest at ec968dc8-2e34-4307-8d3a-9bbffdf9e6cc/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc5a90.so found manifest at d7cc5876-aa06-45b6-9873-513959d3375f/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc93e0.so found manifest at a1c24125-22ab-4b5b-aeeb-4f6550d7b40b/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bce500.so found manifest at 3a08ef80-03d6-4082-9dde-a001f8ba601b/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c1cab0.so found manifest at f30ddcc6-7853-4282-8b0a-305d18827f03/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c2aea0.so found manifest at e946ff05-6c77-4800-9e7e-3d246d48c74c/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c39a60.so found manifest at 5bbf03d5-dd78-4505-85b0-7db70db41870/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c420c0.so found manifest at d50ac8b1-0166-44a6-8499-5d8ecf7c8527/manifest.json (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c4aa80.so [PROC][3596605][ERR] 2021-05-03 07:54:38,179 [INFO][dreamview] Starting Localization module... [PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Perception module... [PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Transform module... [PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Routing module... [PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Prediction module... [PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Planning module... [PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Traffic Light module... [PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Control module... [PROC][3596605][ERR] 2021-05-03 07:54:38,182 [INFO][dreamview] Starting Recorder module... [ANMGR] Initializing with TestReportId:45e0b255-3a6d-493a-b96c-7a7c3ee7bd17 [PROC][3596605][ERR] 2021-05-03 07:54:40,437 [INFO][dreamview] Control message received [PROC][3596605][ERR] 2021-05-03 07:54:42,791 [INFO][run_vse] Starting scenario... [PROC][3596605][ERR] 2021-05-03 07:54:55,148 [INFO][run_vse] All agents traversed their waypoints. [PROC][3596605][ERR] 2021-05-03 07:54:55,149 [INFO][run_vse] Stopping simulation [PROC][3596605][ERR] 2021-05-03 07:54:55,149 [INFO][run_vse] Scenario simulation ended. 5/3/2021 3:54:55 PM|Fatal|<>cDisplayClass174_0.b2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read. at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0 at WebSocketSharp.WebSocketFrame+<>cDisplayClass73_0.b0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0 at WebSocketSharp.Ext+<>c__DisplayClass59_0.b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3596605][ERR] [PROC][3596605][OUT] [PROC][main] Process #3596605 exited with result 0 (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 534 bytes error: 1300 bytes) [LOADER] Stopping simulation on TestCase process exit [LOADER] Cleanup volumes on TestCase process exit [TC] Removing volumes for simulationId:ca9e9997-9f44-48b7-9956-b5f3dbbaa732 [LOADER] Update simulation status Running -> Stopping [CONN] Updated simulation ca9e9997-9f44-48b7-9956-b5f3dbbaa732 with status:Stopping msg: [CONN] POST /api/v1/simulations/ca9e9997-9f44-48b7-9956-b5f3dbbaa732/status Deinitialized the Simulation Network data. (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3596605] Process is already exited Unloading 2 Unused Serialized files (Serialized files now loaded: 17) [ANMGR] Sending test report data id:45e0b255-3a6d-493a-b96c-7a7c3ee7bd17 [CONN] POST /api/v1/test-results/45e0b255-3a6d-493a-b96c-7a7c3ee7bd17 Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32 Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32.so Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/Kernel32 Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32 Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32.so Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32 UnloadTime: 150.756305 ms [LOADER] Update simulation status Stopping -> Idle [CONN] Updated simulation ca9e9997-9f44-48b7-9956-b5f3dbbaa732 with status:Idle msg: [CONN] POST /api/v1/simulations/ca9e9997-9f44-48b7-9956-b5f3dbbaa732/status [CONN] HTTP 200 OK Receiving response of OK System.Net.Http.StreamContent (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Unloading 837 unused Assets to reduce memory usage. Loaded Objects now: 6698. Total: 45.491265 ms (FindLiveObjects: 0.895352 ms CreateObjectMapping: 0.675044 ms MarkObjects: 33.920383 ms DeleteObjects: 9.999193 ms)

Unloading 1 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6687. Total: 37.149180 ms (FindLiveObjects: 0.870221 ms CreateObjectMapping: 0.701224 ms MarkObjects: 35.553388 ms DeleteObjects: 0.023519 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0) [CONN] HTTP 200 OK Receiving response of OK System.Net.Http.StreamContent (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[CONN] HTTP 200 OK Receiving response of OK System.Net.Http.StreamContent (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35) `

EricBoiseLGSVL commented 3 years ago

Are you running the VSE scenario without connecting to AD stack?

GabrielZZZ commented 3 years ago

I ran it with Apollo Dreamview. I am certain there is a connection that the vehicles in dreamview moves simultaneously with the one in the simulator, and all the functions seems normal except that the perception and traffic light module are turned on automatically even under the modular testing scenario, just as #1370.

EricBoiseLGSVL commented 3 years ago

VSE based simulations only support Apollo 5.0 at the moment as it is based on dreamview-python-api. See here

MisterioRemo commented 3 years ago

@EricBoiseLGSVL Hello, I have a similar problem. I want to run VSE + AutowareAuto. So, I start lgsvl_bridge and binary version of simulator on the same machine. The scenario starts playing, lgsvl_bridge subscribes to topics, but after ~20 seconds the scenario stops unexpectedly. Part of my log file:


[CONN] HTTP 200 OK
InitializeOrResetSwapChain 924x1136 hdr=0 samples=1
WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader Unsupported: 'Shader Graphs/VehicleLightsShader' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
found manifest at 52f29c02-6966-4749-bab1-aa0456735740/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5006ce0.so
found manifest at 328b2afb-78c4-4c86-8a18-62015194a3fe/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50dc280.so
found manifest at 1e531ff0-8ce5-4d26-9dda-bbd7a4e2c930/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5009f30.so
found manifest at aa5e3262-fde7-4445-b86e-0922ed5fedba/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5012270.so
found manifest at 68ddcc86-1b72-447e-8b48-736917e07281/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5022ce0.so
found manifest at e8bb3908-0059-4829-8cc3-4ba07dd915bf/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x503f930.so
found manifest at ec968dc8-2e34-4307-8d3a-9bbffdf9e6cc/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50c5b80.so
found manifest at a1022620-f2a3-4cef-9146-d10eaa445386/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50bf040.so
found manifest at 3a08ef80-03d6-4082-9dde-a001f8ba601b/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50ca510.so
found manifest at 3a9b1c3a-3e5a-4b15-9b6c-3acbc3c937a0/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50f5f10.so
found manifest at d50ac8b1-0166-44a6-8499-5d8ecf7c8527/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50f88c0.so
found manifest at ad643739-d4e3-415a-bdf3-0ebcf4bc2f64/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5109fa0.so
found manifest at f30ddcc6-7853-4282-8b0a-305d18827f03/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x51126a0.so
found manifest at 5bba1420-a7fd-40cc-887d-fe88cc6c49b7/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5118ed0.so
found manifest at d7cc5876-aa06-45b6-9873-513959d3375f/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5129760.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Original exception: [Errno 111] Connection refused
[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Something went wrong with bridge / dreamview connection.
InitializeOrResetSwapChain 1024x768 hdr=0 samples=1
Waypoint timestamps absent or invalid, caluclating timestamps based on speed. 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883][ERR] 2021-05-12 12:33:55,636 [INFO][run_vse] Starting scenario...
[ANMGR] Initializing with TestReportId:4c2f7ab5-0575-4731-8fc4-ed35117e16eb
[PROC][39883][ERR] 2021-05-12 12:34:27,316 [INFO][run_vse] Scenario simulation ended.
5/12/2021 3:34:27 PM|Fatal|<>c__DisplayClass174_0.<startReceiving>b__2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                             at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0 
                             at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0 
                             at WebSocketSharp.Ext+<>c__DisplayClass59_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0  
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883][OUT] 
[PROC][39883][ERR] 
[PROC][main] Process #39883 exited with result 0 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 507 bytes error: 640 bytes)
[LOADER] Stopping simulation on TestCase process exit
[LOADER] Cleanup volumes on TestCase process exit
[TC] Removing volumes for simulationId:032fcedc-5e3b-4523-a3e1-7e9efb7074e1
Deinitialized the Simulation Network data. 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883] Process is already exited
Unloading 3 Unused Serialized files (Serialized files now loaded: 16)
[ANMGR] Sending test report data id:4c2f7ab5-0575-4731-8fc4-ed35117e16eb
[CONN] POST /api/v1/test-results/4c2f7ab5-0575-4731-8fc4-ed35117e16eb
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/Kernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
UnloadTime: 57.145063 ms
[LOADER] Update simulation status Starting -> Idle
[CONN] Updated simulation 032fcedc-5e3b-4523-a3e1-7e9efb7074e1 with status:Idle msg:
[CONN] POST /api/v1/simulations/032fcedc-5e3b-4523-a3e1-7e9efb7074e1/status

Unloading 416 unused Assets to reduce memory usage. Loaded Objects now: 6551.
Total: 14.474155 ms (FindLiveObjects: 0.375678 ms CreateObjectMapping: 0.184801 ms MarkObjects: 11.774025 ms  DeleteObjects: 2.139070 ms)

Unloading 1 Unused Serialized files (Serialized files now loaded: 0)
[CONN] HTTP 500 InternalServerError

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.924347 ms (FindLiveObjects: 0.432112 ms CreateObjectMapping: 0.283634 ms MarkObjects: 13.182957 ms  DeleteObjects: 0.025082 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)
[CONN] HTTP 200 OK
Receiving response of OK System.Net.Http.StreamContent 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.369595 ms (FindLiveObjects: 0.550483 ms CreateObjectMapping: 0.426983 ms MarkObjects: 12.368738 ms  DeleteObjects: 0.022741 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 10.193459 ms (FindLiveObjects: 0.314205 ms CreateObjectMapping: 0.154849 ms MarkObjects: 9.713740 ms  DeleteObjects: 0.010270 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.472160 ms (FindLiveObjects: 0.391940 ms CreateObjectMapping: 0.224416 ms MarkObjects: 12.822638 ms  DeleteObjects: 0.032650 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 10.644788 ms (FindLiveObjects: 0.295699 ms CreateObjectMapping: 0.155582 ms MarkObjects: 10.178879 ms  DeleteObjects: 0.014094 ms)

I try the same scenario but with Apollo 5.0 and it stops with the same error described above.

EricBoiseLGSVL commented 3 years ago

We are looking into this issue and hope to have a solution soon. Adding @KarByc

KarByc commented 3 years ago

@MisterioRemo

Thanks for using our VSERunner. About your issue there is some error with the bridge connection:

[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Original exception: [Errno 111] Connection refused [PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Something went wrong with bridge / dreamview connection.

Currently the VSERunner works only with the Apollo, we are working to support the AutowareAuto.

MisterioRemo commented 3 years ago

@KarByc

Thanks for using our VSERunner. About your issue there is some error with the bridge connection:

Yes, I noticed this error too. But judging by the lgsvl_bridge's log, it works fine. It subscribes to topics when the scenario starts, and unsubscribes when the scenario stops working. But there is one oddity. I can see the topic's message through terminal ros2 topic echo /my_topic. But rviz2 and rqt almost never sees the simulator's topics when it plays scenario. In the lgsvl_bridge project I changed reliability policy from reliable to best effort and this mini problem was resolved.

Currently the VSERunner works only with the Apollo, we are working to support the AutowareAuto.

Thank you, I'm looking forward to it.

yejw5 commented 3 years ago

I have encountered the same error with Apollo 5.0. Look forward to the solution.

EricBoiseLGSVL commented 3 years ago

Do you have the connection refused error? What error do you have @yejw5 ?

yejw5 commented 3 years ago

Do you have the connection refused error? What error do you have @yejw5 ?

Sorry, I found that my logs were different. I have run tests in Ubuntu18.04 + Apollo5.0 + lgsvl2021.1, and got failed with lgsvl Player.log as follows:

[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Transform module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Routing module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Prediction module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Planning module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Traffic Light module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Control module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Recorder module...
[ANMGR] Initializing with TestReportId:a6263248-16dc-4f65-bd7c-e3a951e34f18
[PROC][3587][ERR] 2021-05-18 02:18:14,153 [INFO][dreamview] Control message received
[PROC][3587][ERR] 2021-05-18 02:18:15,300 [INFO][run_vse] Starting scenario...
[PROC][3587][ERR] 2021-05-18 02:18:35,314 [INFO][run_vse] Scenario simulation ended.
2021/5/18 10:18:35|Fatal|<>c__DisplayClass174_0.<startReceiving>b__2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                           at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0
                           at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0
                           at WebSocketSharp.Ext+<>c__DisplayClass59_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3587][OUT]
[PROC][3587][ERR]
[PROC][main] Process #3587 exited with result 0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 534 bytes error: 1166 bytes)
[LOADER] Stopping simulation on TestCase process exit
[LOADER] Cleanup volumes on TestCase process exit
[TC] Removing volumes for simulationId:75063b5b-5eaa-4da2-a75e-6466dbf5eb4e
[LOADER] Update simulation status Running -> Stopping
[CONN] Updated simulation 75063b5b-5eaa-4da2-a75e-6466dbf5eb4e with status:Stopping msg:
[CONN] POST /api/v1/simulations/75063b5b-5eaa-4da2-a75e-6466dbf5eb4e/status
Deinitialized the Simulation Network data.
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3587] Process is already exited
Unloading 2 Unused Serialized files (Serialized files now loaded: 0)
[ANMGR] Sending test report data id:a6263248-16dc-4f65-bd7c-e3a951e34f18
[CONN] POST /api/v1/test-results/a6263248-16dc-4f65-bd7c-e3a951e34f18
UnloadTime: 7.215336 ms
[LOADER] Update simulation status Stopping -> Idle
[CONN] Updated simulation 75063b5b-5eaa-4da2-a75e-6466dbf5eb4e with status:Idle msg:
[CONN] POST /api/v1/simulations/75063b5b-5eaa-4da2-a75e-6466dbf5eb4e/status

Besides, the apollo bridge.sh prints:

I0518 10:17:59.558449  3534 global_data.cc:150] [cyber_bridge] host ip: X.X.X.X
E0518 10:18:36.339259  3534 client.cc:98] [cyber_bridge] Client write failed, disconnectingsystem:9
EricBoiseLGSVL commented 3 years ago

We have a solution for this issue and will be releasing a fix soon :)

yejw5 commented 3 years ago

We have a solution for this issue and will be releasing a fix soon :)

Thank you, I'm looking forward to it.

QZJGeorge commented 3 years ago

Same issue here. The simulation shuts down for no reason at about 20 seconds. The bridge did not report an error. Looking forward to a fix.

safdarzareef commented 2 years ago

Hi @EricBoiseLGSVL we are still running into a similar issue, even though Autoware AI connects (2D nav goal and moving the ego vehicle works) when we run a modified Python API test case, Red Light Runner for Apollo, and change the ego vehicle for Autoware AI to Jaguar2015XE, the exception is raised that the AD stack is not ready yet. Is there any fix for this?