GM Volt: Perpetual "Service Front Camera" error on dash

[twilsonco]

Describe the bug

Recently, the "Service Front Camera" error on my Volt has been showing constantly on master or updated forks. Talked about this with @sshane.

I searched back and found the commit where it started, but I haven't yet looked into the actual cause of the problem. I'm opening this issue since you might have a better idea. I'll look into it more as well since it may not cause problems in any other vehicle.

Below are my notes with the problematic commit by @briskspirit:

Going through releases to see where problem started:

Checking for "Service Front Camera" dash error

Version	Date	Commit	Error
0.9.0	2022/11/18	da5d08d	Yes
0.8.17	2022/10/07	6ce511c	Yes
2022/09/15	5a80fda	Yes
2022/09/13	4d35142	Yes
2022/09/13	84a3c355e5fdb7bb2f50847f925e894eecfa8918	Yes
2022/09/12	f1a2dc3	Yes
722e464	No
2022/09/12	29f9c53	No
2022/09/09	aea1a0d	No
0.8.17	2022/09/02	3b602e2	No
0.8.16	2022/08/23	f41dc62	No
0.8.15	2022/07	5a7c2f9	No
0.8.12	2021/12	a965de3	No

The error starts at f1a2dc3, and does not occur for the preceding commit

commit f1a2dc36ba15ce374ef13f77c460d3100d3ab3b3
Author: Igor Biletskyy <briskspirit@users.noreply.github.com>
Date:   Mon Sep 12 15:08:19 2022 -0700

    bump panda and cereal (#25746)

    bump

:160000 160000 cea51afd6 2335f98bb M    cereal
:160000 160000 8f13ca3f6 15eda6a7c M    panda

commit 722e4641fc5aa28ab0ba2de266d230853d5ac321
Author: Dean Lee <deanlee3@gmail.com>
Date:   Tue Sep 13 01:30:58 2022 +0800

    prime.cc: call update() after QR code changed (#25740)

:100644 100644 a8ceee4ca 1d765d094 M    selfdrive/ui/qt/widgets/prime.cc

Corresponds to the panda change on 9/12. The others are here for context.

commit 15eda6a7c9cf4b0c7b5a7bbde25fcab3ff0a98fc
Author: Igor Biletskyy <briskspirit@users.noreply.github.com>
Date:   Mon Sep 12 14:31:22 2022 -0700

    CAN bus: adjust SJW and time quanta to SAE 2284-3 (#1061)

    * adjust clocks and tq

    * debug console keyboard interrupt

    * MISRA

commit ac21dbe551c7e894f66aded38260370d0bd09d03
Author: Igor Biletskyy <briskspirit@users.noreply.github.com>
Date:   Fri Sep 9 20:13:06 2022 -0700

    H7: add chiplet panda support (RPv2) (#1049)

    * temporarily disable Jenkins

    * firmware

    * python lib

    * tests

    * ..

    * usb load switch to high

    * it is still red

    * usb power mode remove

    * Revert "temporarily disable Jenkins"

    This reverts commit a911cfad8daf10763489b7e7bc10107e77eef4bf.

    * forgot this

    * common periph

    * change name

    * board tick

    * add V2 to CI

    * update known bootstub

commit 1b210a8974442e2cc9520f791a97220ebbf71d5f
Author: Igor Biletskyy <briskspirit@users.noreply.github.com>
Date:   Fri Sep 9 15:24:19 2022 -0700

    H7: remove manual bus off handling (#1062)

    remove this

and these cereal changes (these are the most recent by Igor and one related)

commit 2335f98bbe628ec6fde92c8d929ecaf373b125af
Author: Igor Biletskyy <briskspirit@users.noreply.github.com>
Date:   Fri Sep 9 20:13:25 2022 -0700

    add RP with chiplet (#358)

    * chiplet

    * Update log.capnp

    Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

    Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

:100644 100644 aeb7f0d b937ddc M        log.capnp

commit 04b1cf2d6ff326e15b83f2e852e62aa007fa3c49
Author: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Date:   Fri Sep 9 16:49:12 2022 -0700

    includes tape reel

:100644 100644 638403d aeb7f0d M        log.capnp

Which car does this affect?

Chevy Volt 2018

Provide a route where the issue occurs

c11fcb510a549332|2023-02-22--18-27-28--0

openpilot version

0.9.0

Additional info

No response

[sshane]

@briskspirit do you want to check this out? @twilsonco try to revert the panda quanta change and see if it's fixed

[twilsonco]

@sshane The panda quanta change is the one that directly corresponds with the openpilot bump commit where the problem started. I provided the details of the preceding panda commit by Igor just to give you two more context. The preceding commit where it works only differs by that panda/cereal bump, and it's nearly inconceivable that the cereal bump would be the problem.

[twilsonco]

To follow up @sshane, my initial testing including running op from immediately before and after that panda change, which is what you're asking right? No error on https://github.com/commaai/openpilot/commit/722e4641fc5aa28ab0ba2de266d230853d5ac321, yes error on the subsequent https://github.com/commaai/openpilot/commit/f1a2dc36ba15ce374ef13f77c460d3100d3ab3b3, which only includes the panda/cereal changes.

I still haven't looked into this further.

[sshane]

I meant to just cherry pick the quanta panda commit on top of https://github.com/commaai/openpilot/commit/722e4641fc5aa28ab0ba2de266d230853d5ac321 to be sure which submodule commit might be causing this

[twilsonco]

So revert to before that commit, cherry-pick the panda only, and confirm it causes it. Yeah I can do that, and maybe isolate the issue while I’m at it.

On Mar 10, 2023 at 3:32 PM -0700, Shane Smiskol @.***>, wrote:

I meant to just cherry pick the quanta panda commit on 722e464 to be sure — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[sshane]

Did you ever see if reverting the time quanta changes fixed this, or does this not happen anymore @twilsonco?

[twilsonco]

No and yes it still shows the error consistently, and sorry :)

[sshane]

@twilsonco have you tried reverting the panda commit yet to see if that's the issue?

[twilsonco]

OK finally got around to testing f1a2dc3 but with the time quanta change reverted. It does resolve the dash error @sshane. Here's a route of the error not happening (c11fcb510a549332|2024-01-20--18-33-08, not that the dash error shows up in the logs anyhow AFAIK), driven on this branch with that reversion as the only difference from f1a2dc3.