search

commaai / openpilot

openpilot is an operating system for robotics. Currently, it upgrades the driver assistance system in 275+ supported cars.
https://comma.ai/openpilot
MIT License
49.11k stars 8.94k forks source link

2017 CRV Touring - Random ACC error followed by Comma 3 reboot. #22907

Closed kurtnag closed 2 years ago

kurtnag commented 2 years ago

Describe the bug

Adaptive Cruise Control Malfunction on dash followed by Comma 3 Reboot.

connect.comma.ai shows Yaddo as the town name when it crashed and when it resumed.

What hardware does this issue affect?

comma three

Provide a route where the issue occurs

593ebfcbfb2108e7|2021-11-14--13-40-16

openpilot version

0.8.10

Additional info

No response

kurtnag commented 2 years ago

image

cydia2020 commented 2 years ago

Sounds like a connection issue, check your harness and cable

pd0wm commented 2 years ago

Looking at the pstore (the last recorded kernel messages from the previous boot), there was a kernel panic related to accessing the flash that caused the reboot. I need to do some more investigation to see if this is a hardware or software issue. Please update this issue if it happens again.

Snippet of the dump:

[ 1232.035498] ufs_qcom_phy_qmp_v3 1d87000.ufsphy_mem: ufs_qcom_phy_qmp_v3_is_pcs_ready: poll for pcs failed err = -110
[ 1232.035533] ufshcd-qcom 1d84000.ufshc: ufs_qcom_power_up_sequence: is_physical_coding_sublayer_ready() failed, ret = -110
[ 1232.249793] ufshcd-qcom 1d84000.ufshc: Controller enable failed
[ 1232.249827] ufshcd-qcom 1d84000.ufshc: ufshcd_host_reset_and_restore: Host init failed -5
[ 1232.673300] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 11 old ctx ffffff8d220018e8 cookie 23822 new ctx ffffff8d22006128 cookie 23835
[ 1233.252721] ufs_qcom_phy_qmp_v3 1d87000.ufsphy_mem: ufs_qcom_phy_qmp_v3_is_pcs_ready: poll for pcs failed err = -110
[ 1233.252754] ufshcd-qcom 1d84000.ufshc: ufs_qcom_power_up_sequence: is_physical_coding_sublayer_ready() failed, ret = -110
[ 1233.469909] ufshcd-qcom 1d84000.ufshc: Controller enable failed
[ 1233.469943] ufshcd-qcom 1d84000.ufshc: ufshcd_host_reset_and_restore: Host init failed -5
[ 1233.623276] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 38 old ctx ffffff8d22006128 cookie 23841 new ctx ffffff8d2200a968 cookie 23854
[ 1234.472923] ufs_qcom_phy_qmp_v3 1d87000.ufsphy_mem: ufs_qcom_phy_qmp_v3_is_pcs_ready: poll for pcs failed err = -110
[ 1234.472962] ufshcd-qcom 1d84000.ufshc: ufs_qcom_power_up_sequence: is_physical_coding_sublayer_ready() failed, ret = -110
[ 1234.699837] ufshcd-qcom 1d84000.ufshc: Controller enable failed
[ 1234.699869] ufshcd-qcom 1d84000.ufshc: ufshcd_host_reset_and_restore: Host init failed -5
[ 1235.173273] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 59 old ctx ffffff8d22006128 cookie 23872 new ctx ffffff8d2200a968 cookie 23885
[ 1235.702656] ufs_qcom_phy_qmp_v3 1d87000.ufsphy_mem: ufs_qcom_phy_qmp_v3_is_pcs_ready: poll for pcs failed err = -110
[ 1235.702691] ufshcd-qcom 1d84000.ufshc: ufs_qcom_power_up_sequence: is_physical_coding_sublayer_ready() failed, ret = -110
[ 1235.919852] ufshcd-qcom 1d84000.ufshc: Controller enable failed
[ 1235.919886] ufshcd-qcom 1d84000.ufshc: ufshcd_host_reset_and_restore: Host init failed -5
[ 1236.922846] ufs_qcom_phy_qmp_v3 1d87000.ufsphy_mem: ufs_qcom_phy_qmp_v3_is_pcs_ready: poll for pcs failed err = -110
[ 1236.922882] ufshcd-qcom 1d84000.ufshc: ufs_qcom_power_up_sequence: is_physical_coding_sublayer_ready() failed, ret = -110
[ 1237.139954] ufshcd-qcom 1d84000.ufshc: Controller enable failed
[ 1237.139989] ufshcd-qcom 1d84000.ufshc: ufshcd_host_reset_and_restore: Host init failed -5
[ 1237.140036] Kernel BUG at ffffff8d1f8858e0 [verbose debug info unavailable]
[ 1237.140067] ------------[ cut here ]------------
[ 1237.140080] Kernel BUG at ffffff8d1f8858e0 [verbose debug info unavailable]
[ 1237.140094] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 1237.140110] Modules linked in: wlan(CE) snd_soc_sdm845(E) snd_soc_wcd9xxx(E)
[ 1237.140146] CPU: 0 PID: 4 Comm: kworker/0:0 Tainted: G         C  E   4.9.103+ #1
[ 1237.140160] Hardware name: Qualcomm Technologies, Inc. sda845 v2.1 TurboX-SOM_V01 (DT)
[ 1237.140183] Workqueue: events ufshcd_err_handler
[ 1237.140204] task: ffffffd67528aa00 task.stack: ffffffd6752b0000
[ 1237.140220] PC is at ufshcd_reset_and_restore.part.50+0x10/0x18
[ 1237.140235] LR is at ufshcd_reset_and_restore+0x8c/0x90
[ 1237.140248] pc : [<ffffff8d1f8858e0>] lr : [<ffffff8d1f890ca4>] pstate: 80c00145
[ 1237.140261] sp : ffffffd6752b3ca0
[ 1237.140273] x29: ffffffd6752b3ca0 x28: 0000000000000000 
[ 1237.140303] x27: 0000000000000000 x26: ffffff8d21662000 
[ 1237.140332] x25: ffffffd66ce327e0 x24: 0000000000000000 
[ 1237.140361] x23: 0000000000000000 x22: 00000000ffffffff 
[ 1237.140390] x21: 00000000fffffffb x20: ffffffd66ce327e0 
[ 1237.140419] x19: 0000000000000000 x18: 0000000000000010 
[ 1237.140449] x17: 000000000057a34d x16: ffffff8d21705ac8 
[ 1237.140478] x15: ffffffffffffffff x14: 3a65726f74736572 
[ 1237.140508] x13: 5f646e615f746573 x12: 65725f74736f685f 
[ 1237.140537] x11: 00000000000013e4 x10: 63687366752e3030 
[ 1237.140565] x9 : 0000000000003ff0 x8 : 0000000000003fff 
[ 1237.140594] x7 : ffffffd67dc3f090 x6 : 000000000a7d4b21 
[ 1237.140625] x5 : 00ffffffffffffff x4 : ffffff8d21479000 
[ 1237.140655] x3 : ffffff8d21479628 x2 : d39525cff5e87700 
[ 1237.140684] x1 : d39525cff5e87700 x0 : ffffff8d1f890ca4 
[ 1237.140714] 
[ 1237.140714] PC: 0xffffff8d1f8858a0:
[ 1237.140728] 58a0  90005482 913aa042 2a0403f3 aa1403e0 910e0042 90007bc1 9130e021 97fcf9b3
[ 1237.140823] 58c0  2a1303e0 a94153f3 a8c27bfd d65f03c0 a9bf7bfd 910003fd aa1e03e0 d503201f
[ 1237.140916] 58e0  d4210000 d503201f d10103ff a9017bfd 910043fd a90253f3 a9035bf5 aa0003f4
[ 1237.141008] 5900  aa0103f5 aa1e03e0 12001c56 d503201f f9400293 52800020 9100c273 aa1303e1
[ 1237.141102] 
[ 1237.141102] LR: 0xffffff8d1f890c64:
[ 1237.141115] 0c64  9426d669 aa0003f3 aa1403e0 97ffd8f8 aa1403e0 97ffc3f8 f9401e80 aa1303e1
[ 1237.141208] 0c84  f9402c00 9426d764 2a1503e0 a94153f3 f94013f5 a8c37bfd d65f03c0 97ffd30c
[ 1237.141301] 0ca4  d503201f a9bc7bfd 910003fd a90153f3 a9025bf5 f9001bf7 aa0003f3 2a0103f6
[ 1237.141393] 0cc4  aa1e03e0 52800035 d503201f b900a275 aa1303e0 b9405677 97ffd025 2a1503e1
[ 1237.141486] 
[ 1237.141486] SP: 0xffffffd6752b3c60:
[ 1237.141499] 3c60  1f890ca4 ffffff8d 752b3ca0 ffffffd6 1f8858e0 ffffff8d 80c00145 00000000
[ 1237.141591] 3c80  fffffffb 00000000 ffffffff 00000000 ffffffff ffffffff 1f890c40 ffffff8d
[ 1237.141683] 3ca0  752b3cb0 ffffffd6 1f890ca4 ffffff8d 752b3ce0 ffffffd6 1f8916a8 ffffff8d
[ 1237.141775] 3cc0  6ce32960 ffffffd6 21478788 ffffff8d 00000140 00000000 21478788 ffffff8d
[ 1237.141869] Process kworker/0:0 (pid: 4, stack limit = 0xffffffd6752b0000)
[ 1237.141883] Call trace:
[ 1237.141897] Exception stack(0xffffffd6752b3aa0 to 0xffffffd6752b3bd0)
[ 1237.141912] 3aa0: 0000000000000000 0000007fffffffff ffffffd6752b3ca0 ffffff8d1f8858e0
[ 1237.141927] 3ac0: 0000000080c00145 000000000000003d ffffffd66ce327e0 d39525cff5e87700
[ 1237.141941] 3ae0: ffffffd6752b3b90 ffffff8d1f7c3ce4 ffffffd673ef2c10 ffffff8d2071fe10
[ 1237.141955] 3b00: ffffffd6752b3bf8 0000000000000140 ffffffd6752b3b90 ffffffd6752b3b90
[ 1237.141969] 3b20: ffffffd6752b3b60 00000000ffffffd8 ffffffd6752b3bd8 ffffffd6752b3b90
[ 1237.141984] 3b40: ffffffd6752b3b90 ffffffd6752b3b60 00000000ffffffd8 d39525cff5e87700
[ 1237.141998] 3b60: ffffffd6752b3b90 d39525cff5e87700 ffffff8d1f890ca4 d39525cff5e87700
[ 1237.142012] 3b80: d39525cff5e87700 ffffff8d21479628 ffffff8d21479000 00ffffffffffffff
[ 1237.142027] 3ba0: 000000000a7d4b21 ffffffd67dc3f090 0000000000003fff 0000000000003ff0
[ 1237.142040] 3bc0: 63687366752e3030 00000000000013e4
[ 1237.142054] [<ffffff8d1f8858e0>] ufshcd_reset_and_restore.part.50+0x10/0x18
[ 1237.142069] [<ffffff8d1f890ca4>] ufshcd_reset_and_restore+0x8c/0x90
[ 1237.142083] [<ffffff8d1f8916a8>] ufshcd_err_handler+0x1f8/0x810
[ 1237.142101] [<ffffff8d1f0cb6bc>] process_one_work+0x20c/0x4d8
[ 1237.142116] [<ffffff8d1f0cb9d8>] worker_thread+0x50/0x4d0
[ 1237.142131] [<ffffff8d1f0d2980>] kthread+0x100/0x108
[ 1237.142147] [<ffffff8d1f083f00>] ret_from_fork+0x10/0x50
[ 1237.142163] Code: a9bf7bfd 910003fd aa1e03e0 d503201f (d4210000) 
[ 1237.142184] ---[ end trace f79761bdb1e40cbd ]---
kurtnag commented 2 years ago

Can you also look at route 593ebfcbfb2108e7|2021-11-11--12-52-38

It stopped recording while OP was active.

Also for my own education, what file are you looking at for this info?

Thanks!

kurtnag commented 2 years ago

It's happening all the time now. My first route today went about 30 min then it reboot and came back. The subsequent routes did not reboot, just went black. The blue light is solid when it goes black, not sure what it is when it reboots.

Each time I get a ACC malfunction, look up and the image on the screen is frozen.

image

ffhspa commented 2 years ago

I get the same annoying Error cruiseMismatch this never was an issue before. I use Honda Civic Bosch Diesel

kurtnag commented 2 years ago

I sent an email to support since it's no longer random.

I'm also looking at a new Toyota so my debugging may get delayed. Also have a replacement USB C cable on the way from Amazon that Erich recommends on Discord.

kurtnag commented 2 years ago

Support is having me send my unit back. Closing