Manoj-EK
commented
3 years ago @ilovevision you can look at the issue #12998 .............it might help
Closed ilovevision closed 2 years ago
Manoj-EK
commented
3 years ago @ilovevision you can look at the issue #12998 .............it might help
daohu527
commented
2 years ago Closed due to inactivity. If the problem persists, pls feel free to reopen it or create a new one and refer to it.
We appreciate you go through Apollo documentations and search previous issues before creating an new one. If neither of the sources helped you with your issues, please report the issue using the following form. Please note missing info can delay the response time.
System information
Steps to reproduce the issue:
Supporting materials (screenshots, command lines, code/script snippets):
I couldn't open the module of perception on the dreamview, so I tried to open it by a new terminal in th docker.
bash scripts/perception.sh
[cyber_launch_10770] INFO Stop cyber launch finished. [cyber_launch_10770] INFO cyber_launch exit. [cyber_launch_10805] INFO Stop cyber launch finished. [cyber_launch_10805] INFO cyber_launch exit. [cyber_launch_10853] INFO Stop cyber launch finished. [cyber_launch_10853] INFO cyber_launch exit. [catarc@in-dev-docker:/apollo]$ [cyber_launch_10888] INFO Launch file [/apollo/modules/transform/launch/static_transform.launch] [cyber_launch_10888] INFO ======================================================================================================================== [cyber_launch_10888] INFO Load module [static_transform] library: [mainboard_default_10888] [CYBER_DEFAULT] conf: [/apollo/modules/transform/dag/static_transform.dag] exception_handler: [] [cyber_launch_10888] INFO Start process [mainboard_default_10888] successfully. pid: 10911 [cyber_launch_10888] INFO ------------------------------------------------------------------------------------------------------------------------ [cyber_launch_10890] INFO Launch file [/apollo/modules/perception/production/launch/perception_all.launch] [cyber_launch_10890] INFO ======================================================================================================================== [cyber_launch_10890] INFO Load module [perception] library: [perception] [CYBER_DEFAULT] conf: [/apollo/modules/perception/production/dag/dag_streaming_perception.dag] exception_handler: [] [cyber_launch_10890] INFO Start process [perception] successfully. pid: 10913 [cyber_launch_10890] INFO ------------------------------------------------------------------------------------------------------------------------ [cyber_launch_10890] INFO Load module [perception_camera] library: [perception] [CYBER_DEFAULT] conf: [/apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag] exception_handler: [] [cyber_launch_10890] INFO Load module [perception_traffic_light] library: [perception_trafficlights] [CYBER_DEFAULT] conf: [/apollo/modules/perception/production/dag/dag_streaming_perception_trafficlights.dag] exception_handler: [] [cyber_launch_10890] INFO Start process [perception_trafficlights] successfully. pid: 10915 [cyber_launch_10890] INFO ------------------------------------------------------------------------------------------------------------------------ [cyber_launch_10890] INFO Load module [motion_service] library: [motion_service] [CYBER_DEFAULT] conf: [/apollo/modules/perception/production/dag/dag_motion_service.dag] exception_handler: [] [cyber_launch_10889] INFO Launch file [/apollo/modules/drivers/tools/image_decompress/launch/image_decompress.launch] [cyber_launch_10889] INFO ======================================================================================================================== [cyber_launch_10889] INFO Load module [image_decompress] library: [mainboard_default_10889] [CYBER_DEFAULT] conf: [/apollo/modules/drivers/tools/image_decompress/dag/image_decompress.dag] exception_handler: [] [cyber_launch_10889] INFO Start process [mainboard_default_10889] successfully. pid: 10918 [cyber_launch_10889] INFO ------------------------------------------------------------------------------------------------------------------------ [cyber_launch_10890] INFO Start process [motion_service] successfully. pid: 10917 [cyber_launch_10890] INFO ------------------------------------------------------------------------------------------------------------------------ [perception_trafficlights] WARNING: Logging before InitGoogleLogging() is written to STDERR [perception_trafficlights] I1120 16:53:49.833937 10915 module_argument.cc:81] []command: mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception_trafficlights.dag -p perception_trafficlights -s CYBER_DEFAULT [perception_trafficlights] I1120 16:53:49.834486 10915 global_data.cc:153] []host ip: 192.168.1.54 [perception_trafficlights] I1120 16:53:49.835630 10915 module_argument.cc:57] []binaryname is mainboard, processgroup is perception_trafficlights, has 1 dag conf [perception_trafficlights] I1120 16:53:49.835645 10915 module_argument.cc:60] []dag_conf: /apollo/modules/perception/production/dag/dag_streaming_perception_trafficlights.dag [perception] WARNING: Logging before InitGoogleLogging() is written to STDERR [perception] I1120 16:53:49.840821 10913 module_argument.cc:81] []command: mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception.dag /apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag -p perception -s CYBER_DEFAULT [perception] I1120 16:53:49.840931 10913 module_argument.cc:120] []Found non-option ARGV-element "/apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag" [perception] I1120 16:53:49.840935 10913 module_argument.cc:29] []Usage: [perception] mainboard [OPTION]... [perception] Description: [perception] -h, --help : help information [perception] -d, --dag_conf=CONFIG_FILE : module dag config file [perception] -p, --process_group=process_group: the process namespace for running this module, default in manager process [perception] -s, --sched_name=sched_name: sched policy conf for hole process, sched_name should be conf in cyber.pb.conf [perception] Example: [perception] mainboard -h [perception] mainboard -d dag_conf_file1 -d dag_conf_file2 -p process_group -s sched_name [mainboard_default_10888] WARNING: Logging before InitGoogleLogging() is written to STDERR [mainboard_default_10888] I1120 16:53:49.859946 10911 module_argument.cc:81] []command: mainboard -d /apollo/modules/transform/dag/static_transform.dag -p mainboard_default_10888 -s CYBER_DEFAULT [mainboard_default_10888] I1120 16:53:49.860452 10911 global_data.cc:153] []host ip: 192.168.1.54 [mainboard_default_10888] I1120 16:53:49.861234 10911 module_argument.cc:57] []binaryname is mainboard, processgroup is mainboard_default_10888, has 1 dag conf [mainboard_default_10888] I1120 16:53:49.861246 10911 module_argument.cc:60] []dag_conf: /apollo/modules/transform/dag/static_transform.dag [mainboard_default_10889] WARNING: Logging before InitGoogleLogging() is written to STDERR [mainboard_default_10889] I1120 16:53:49.862565 10918 module_argument.cc:81] []command: mainboard -d /apollo/modules/drivers/tools/image_decompress/dag/image_decompress.dag -p mainboard_default_10889 -s CYBER_DEFAULT [mainboard_default_10889] I1120 16:53:49.863018 10918 global_data.cc:153] []host ip: 192.168.1.54 [mainboard_default_10889] I1120 16:53:49.863798 10918 module_argument.cc:57] []binaryname is mainboard, processgroup is mainboard_default_10889, has 1 dag conf [mainboard_default_10889] I1120 16:53:49.863809 10918 module_argument.cc:60] []dag_conf: /apollo/modules/drivers/tools/image_decompress/dag/image_decompress.dag [motion_service] WARNING: Logging before InitGoogleLogging() is written to STDERR [motion_service] I1120 16:53:49.883980 10917 module_argument.cc:81] []command: mainboard -d /apollo/modules/perception/production/dag/dag_motion_service.dag -p motion_service -s CYBER_DEFAULT [motion_service] I1120 16:53:49.884368 10917 global_data.cc:153] []host ip: 192.168.1.54 [motion_service] I1120 16:53:49.884989 10917 module_argument.cc:57] []binaryname is mainboard, processgroup is motion_service, has 1 dag conf [motion_service] I1120 16:53:49.884999 10917 module_argument.cc:60] []dag_conf: /apollo/modules/perception/production/dag/dag_motion_service.dag [motion_service] E1120 16:53:49.930346 10917 module_controller.cc:87] [mainboard]Path does not exist: /apollo/bazel-bin/modules/perception/camera/lib/motion_service/libmotion_service.so [motion_service] E1120 16:53:49.930413 10917 module_controller.cc:67] [mainboard]Failed to load module: /apollo/modules/perception/production/dag/dag_motion_service.dag [motion_service] E1120 16:53:49.930425 10917 mainboard.cc:39] [mainboard]module start error. [mainboard_default_10888] E1120 16:53:49.957656 10911 static_transform_component.cc:60] [static_transform]Extrinsic yaml file does not exist: /apollo/modules/drivers/velodyne/params/lidar16_front_up_novatel_extrinsics.yaml [mainboard_default_10888] E1120 16:53:49.957706 10911 static_transform_component.cc:60] [static_transform]Extrinsic yaml file does not exist: /apollo/modules/drivers/velodyne/params/lidar16_front_novatel_extrinsics.yaml [mainboard_default_10888] E1120 16:53:49.957713 10911 static_transform_component.cc:60] [static_transform]Extrinsic yaml file does not exist: /apollo/modules/drivers/velodyne/params/lidar16_rear_left_novatel_extrinsics.yaml [mainboard_default_10888] E1120 16:53:49.957721 10911 static_transform_component.cc:60] [static_transform]Extrinsic yaml file does not exist: /apollo/modules/drivers/velodyne/params/lidar16_rear_right_novatel_extrinsics.yaml [mainboard_default_10888] E1120 16:53:49.958560 10911 static_transform_component.cc:60] [static_transform]Extrinsic yaml file does not exist: /apollo/modules/perception/data/params/radar_rear_extrinsics.yaml [mainboard_default_10889] E1120 16:53:50.160631 10993 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[camera_front_6mm_decompress], please ensure that each node has a unique name. [mainboard_default_10889] E1120 16:53:50.160699 10993 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[camera_front_12mm_decompress], please ensure that each node has a unique name. [mainboard_default_10889] E1120 16:53:50.160728 10993 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[camera_left_fisheye_decompress], please ensure that each node has a unique name. [mainboard_default_10889] E1120 16:53:50.160750 10993 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[camera_right_fisheye_decompress], please ensure that each node has a unique name. [mainboard_default_10889] E1120 16:53:50.160773 10993 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[camera_rear_6mm_decompress], please ensure that each node has a unique name. [mainboard_default_10888] E1120 16:53:50.167915 11000 node_manager.cc:101] [mainboard]this process will be terminated due to duplicated node[static_transform], please ensure that each node has a unique name. [mainboard_default_10888], std::allocator > const&, void const*) + 0x67 (0x7f07f913f067 in /usr/local/libtorch_gpu/lib/libc10.so)
[perception_trafficlights] 1: + 0x2b1d12d (0x7f079232d12d in /usr/local/libtorch_gpu/lib/libtorch_cuda.so)
[perception_trafficlights] 2: caffe2::CUDAContext::~CUDAContext() + 0x4b (0x7f079232973b in /usr/local/libtorch_gpu/lib/libtorch_cuda.so)
[perception_trafficlights] 3: + 0x2ee7ef6 (0x7f07926f7ef6 in /usr/local/libtorch_gpu/lib/libtorch_cuda.so)
[perception_trafficlights] 4: + 0x2d106bc (0x7f07925206bc in /usr/local/libtorch_gpu/lib/libtorch_cuda.so)
[perception_trafficlights] 5: + 0x2d5610c (0x7f07f639f10c in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 6: + 0x2d11ad7 (0x7f07f635aad7 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 7: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator >&) + 0x35 (0x7f07f634f375 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 8: + 0x2ce4ae2 (0x7f07f632dae2 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 9: torch::jit::GraphFunction::operator()(std::vector<c10::IValue, std::allocator >, std::unordered_map<std::cxx11::basic_string<char, std::char_traits, std::allocator >, c10::IValue, std::hash<std:: cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std:: cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::cxx11::basic_string<char, std::char_traits, std::allocator > const, c10::IValue> > > const&) + 0x3a (0x7f07f665cb2a in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 10: torch::jit::Method::operator()(std::vector<c10::IValue, std::allocator >, std::unordered_map<std:: cxx11::basic_string<char, std::char_traits, std::allocator >, c10::IValue, std::hash<std::cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std:: cxx11::basic_string<char, std::char_traits, std::allocator > const, c10::IValue> > > const&) + 0xe0 (0x7f07f660e950 in /usr/local/libtorch_gpu/lib/libtorch_cpu.so)
[perception_trafficlights] 11: torch::jit::Module::forward(std::vector<c10::IValue, std::allocator >) + 0xff (0x7f084a2aa9d3 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sinference_Slibtorch_Slibtorch_Udet.so)
[perception_trafficlights] 12: apollo::perception::inference::TorchDet::Infer() + 0x52a (0x7f084a2a13c8 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sinference_Slibtorch_Slibtorch_Udet.so)
[perception_trafficlights] 13: apollo::perception::camera::TrafficLightDetection::Inference(std::vector<std::shared_ptr, std::allocator<std::shared_ptr > >, apollo::perception::camera::DataProvider) + 0x951 (0x7f084b323211 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Scamera_Slib_Straffic_Ulight_Sdetector_Sdetection_Slibdetection.so)
[perception_trafficlights] 14: apollo::perception::camera::TrafficLightDetection::Detect(apollo::perception::camera::TrafficLightDetectorOptions const&, apollo::perception::camera::CameraFrame) + 0x556 (0x7f084b3239ec in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Scamera_Slib_Straffic_Ulight_Sdetector_Sdetection_Slibdetection.so)
[perception_trafficlights] 15: apollo::perception::camera::TrafficLightCameraPerception::Perception(apollo::perception::camera::CameraPerceptionOptions const&, apollo::perception::camera::CameraFrame) + 0x4b (0x7f08683bc2e7 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Scamera_Sapp_Slibtraffic_Ulight_Ucamera_Uperception.so)
[perception_trafficlights] 16: apollo::perception::onboard::TrafficLightsPerceptionComponent::OnReceiveImage(std::shared_ptr, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xd10 (0x7f084bc283f0 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 17: void std:: invoke_impl<void, void (apollo::perception::onboard::TrafficLightsPerceptionComponent::&)(std::shared_ptr, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&), apollo::perception::onboard::TrafficLightsPerceptionComponent &, std::shared_ptr const&, std:: cxx11::basic_string<char, std::char_traits, std::allocator >&>(std::invoke_memfun_deref, void (apollo::perception::onboard::TrafficLightsPerceptionComponent::&)(std::shared_ptr, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&), apollo::perception::onboard::TrafficLightsPerceptionComponent &, std::shared_ptr const&, std:: cxx11::basic_string<char, std::char_traits, std::allocator >&) + 0xab (0x7f084bc79657 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 18: std::invoke_result<void (apollo::perception::onboard::TrafficLightsPerceptionComponent::*&)(std::shared_ptr, std:: cxx11::basic_string<char, std::char_traits, std::allocator > const&), apollo::perception::onboard::TrafficLightsPerceptionComponent&, std::shared_ptr const&, std::cxx11::basic_string<char, std::char_traits, std::allocator >&>::type std:: invoke<void (apollo::perception::onboard::TrafficLightsPerceptionComponent:: &)(std::shared_ptr, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&), apollo::perception::onboard::TrafficLightsPerceptionComponent&, std::shared_ptr const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&>(void (apollo::perception::onboard::TrafficLightsPerceptionComponent:: &)(std::shared_ptr, std:: cxx11::basic_string<char, std::char_traits, std::allocator > const&), apollo::perception::onboard::TrafficLightsPerceptionComponent&, std::shared_ptr const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) + 0x6f (0x7f084bc723e9 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 19: void std::_Bind<void (apollo::perception::onboard::TrafficLightsPerceptionComponent:: (apollo::perception::onboard::TrafficLightsPerceptionComponent, std::_Placeholder<1>, std::__cxx11::basic_string<char, std::char_traits, std::allocator >))(std::shared_ptr, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&)>:: call<void, std::shared_ptr const&, 0ul, 1ul, 2ul>(std::tuple<std::shared_ptr const&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) + 0xa0 (0x7f084bc685b8 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 20: void std::_Bind<void (apollo::perception::onboard::TrafficLightsPerceptionComponent:: (apollo::perception::onboard::TrafficLightsPerceptionComponent, std::_Placeholder<1>, std::__cxx11::basic_string<char, std::char_traits, std::allocator >))(std::shared_ptr, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>::operator()<std::shared_ptr const&, void>(std::shared_ptr const&) + 0x47 (0x7f084bc5b4bb in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 21: std::_Function_handler<void (std::shared_ptr const&), std::_Bind<void (apollo::perception::onboard::TrafficLightsPerceptionComponent:: (apollo::perception::onboard::TrafficLightsPerceptionComponent*, std::_Placeholder<1>, std::cxx11::basic_string<char, std::char_traits, std::allocator >))(std::shared_ptr, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&)> >::_M_invoke(std::_Any_data const&, std::shared_ptr const&) + 0x37 (0x7f084bc50070 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 22: std::function<void (std::shared_ptr const&)>::operator()(std::shared_ptr const&) const + 0x49 (0x7f084bc727d7 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 1}::operator()(std::shared_ptr const&) const + 0x51 (0x7f084bc68b7d in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 1}>::_M_invoke(std::_Any_data const&, std::shared_ptr const&) + 0x37 (0x7f084bc807f3 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 25: std::function<void (std::shared_ptr const&)>::operator()(std::shared_ptr const&) const + 0x49 (0x7f084bc727d7 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 1}::operator()() const + 0x74 (0x7f084bc72a5c in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 1}>::_M_invoke(std::_Any_data const&) + 0x20 (0x7f084bc91394 in /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_local/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ucamera.so)
[perception_trafficlights] 28: std::function<void ()>::operator()() const + 0x32 (0x7f08a5e90b86 in /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/k8-fastbuild/bin/cyber/../_solib_local/libcyber_Slibinit.so)
[perception_trafficlights] 29: apollo::cyber::croutine::CRoutine::Run() + 0x1c (0x7f089be84152 in /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/k8-fastbuild/bin/cyber/../_solib_local/libcyber_Scroutine_Slibcroutine.so)
[perception_trafficlights] 30: + 0xd5c5 (0x7f089be835c5 in /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/execroot/apollo/bazel-out/k8-fastbuild/bin/cyber/../_solib_local/libcyber_Scroutine_Slibcroutine.so)
[perception_trafficlights] torch/detectron2/export/caffe2_modeling.py", line 16, in forward
[perception_trafficlights] _5, _6, _7, _8, _9, _10, = (_2).forward(input, )
[perception_trafficlights] _11 = (_1).forward(_5, _6, _7, _8, _9, im_info, )
[perception_trafficlights] _12 = (_0).forward(_11, _5, _6, _7, _10, im_info, )
[perception_trafficlights] call
[perception_trafficlights] /home/lishaohua01/Workspace/detection/detectron2/detectron2/modeling/roi_heads/roi_heads.py(727): _forward_box
[perception_trafficlights] /home/lishaohua01/Workspace/detection/detectron2/detectron2/modeling/roi_heads/roi_heads.py(674): forward
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): call
[perception_trafficlights] /home/lishaohua01/Workspace/detection/detectron2/detectron2/export/caffe2_modeling.py(272): forward
[perception_trafficlights] /usr/lib/python3.6/contextlib.py(52): inner
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): call
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/jit/init.py(1027): trace_module
[perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/jit/init__.py(875): trace
[perception_trafficlights] /home/lishaohua01/Workspace/detection/detectron2/detectron2/export/api.py(142): export_torchscript
[perception_trafficlights] tools/deploy/caffe2_converter.py(73):
[perception_trafficlights] RuntimeError: [enforce fail at context_gpu.cu:513] error == cudaSuccess. 2 vs 0. Error at: /home/liufeng/tmp/pytorch/caffe2/core/context_gpu.cu:513: out of memory
[perception_trafficlights]
[mainboard_default_10889]
[cyber_launch_10888] ERROR Process [mainboard_default_10888] has finished. [pid 10911, cmd mainboard -d /apollo/modules/transform/dag/static_transform.dag -p mainboard_default_10888 -s CYBER_DEFAULT]. [cyber_launch_10888] INFO All processes has died. [cyber_launch_10888] INFO Cyber exit. [cyber_launch_10888] INFO All processes have been stopped. [cyber_launch_10889] ERROR Process [mainboard_default_10889] has finished. [pid 10918, cmd mainboard -d /apollo/modules/drivers/tools/image_decompress/dag/image_decompress.dag -p mainboard_default_10889 -s CYBER_DEFAULT]. [cyber_launch_10889] INFO All processes has died. [cyber_launch_10889] INFO Cyber exit. [cyber_launch_10889] INFO All processes have been stopped. [motion_service]
[perception_trafficlights] E1120 16:53:54.007110 10932 tl_preprocessor.cc:125] [mainboard]update_lights_projection failed,lights_outside_image->size() 2 ts: 1.60586e+09 [perception_trafficlights] [E context_gpu.cu:444] Encountered following in ~CUDAContext: [enforce fail at context_gpu.h:198] . Encountered CUDA error: out of memory [perception_trafficlights] 0: c10::ThrowEnforceNotMet(char const, int, char const, std::cxx11::basic_string<char, std::char_traits
[perception_trafficlights] terminate called after throwing an instance of 'std::runtime_error' [perception_trafficlights] what(): The following operation failed in the TorchScript interpreter. [perception_trafficlights] Traceback of TorchScript, serialized code (most recent call last): [perception_trafficlights] File "code/
~~~ <--- HERE [perception_trafficlights] _13, _14, _15, = _12 [perception_trafficlights] return (_13, _14, _15) [perception_trafficlights] File "code/torch/detectron2/modeling/roi_heads/roi_heads.py", line 15, in forward [perception_trafficlights] im_info: Tensor) -> Tuple[Tensor, Tensor, Tensor]: [perception_trafficlights] _0 = self.box_head [perception_trafficlights] _1 = (self.box_pooler).forward(argument_1, argument_2, argument_3, argument_4, argument_5, ) [perception_trafficlights]~~~~~~~~ <--- HERE [perception_trafficlights] _2 = (self.box_predictor).forward((_0).forward(_1, ), ) [perception_trafficlights] _3, _4, = _2 [perception_trafficlights] File "code/torch__/detectron2/export/c10.py", line 55, in forward [perception_trafficlights] roi_fpn2 = ops._caffe2.CopyCPUToGPU(t12, None) [perception_trafficlights] rois_idx_restore_int32 = ops._caffe2.CopyCPUToGPU(t13, None) [perception_trafficlights] _18 = ops._caffe2.RoIAlign(argument_2, roi_fpn, "NCHW", 0.25, 7, 7, 0, True, None) [perception_trafficlights]~~~~<--- HERE [perception_trafficlights] _19 = ops._caffe2.RoIAlign(argument_3, roi_fpn0, "NCHW", 0.125, 7, 7, 0, True, None) [perception_trafficlights] _20 = ops._caffe2.RoIAlign(argument_4, roi_fpn1, "NCHW", 0.0625, 7, 7, 0, True, None) [perception_trafficlights][perception_trafficlights] Traceback of TorchScript, original code (most recent call last): [perception_trafficlights] /home/lishaohua01/Workspace/detection/detectron2/detectron2/export/c10.py(336): forward [perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward [perception_trafficlights] /home/lishaohua01/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548):
[perception_trafficlights]
[cyber_launch_10890] ERROR Process [perception] has died [pid 10913, exit code 1, cmd mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception.dag /apollo/modules/perception/production/dag/dag_streaming_perception_camera.dag -p perception -s CYBER_DEFAULT]. [cyber_launch_10890] ERROR Process [perception_trafficlights] has died [pid 10915, exit code -6, cmd mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception_trafficlights.dag -p perception_trafficlights -s CYBER_DEFAULT]. [cyber_launch_10890] ERROR Process [motion_service] has died [pid 10917, exit code 255, cmd mainboard -d /apollo/modules/perception/production/dag/dag_motion_service.dag -p motion_service -s CYBER_DEFAULT]. [cyber_launch_10890] INFO All processes has died. [cyber_launch_10890] INFO Cyber exit. [cyber_launch_10890] INFO All processes have been stopped.