“Dashcam Mode car unrecognized” occur.

haram-KONA commented 5 months ago

Describe the bug

When turn on the car while the comma C3X is connected to the car, “Dashcam Mode car unrecognized” will almost always occur. reboot the C3X in this state, it will engage normally again. After engaging, drive in openpilot normally and turn off the engine with the C3X connected and then turn it on again, the "Dashcam Mode car unrecognized" status will also occur, and reboot again, the engagement will return to normal. always turn on the ignition without connecting the USB-C of the C3X and then connect the USB-C cable, this problem will not occur. Do I always have to start the car with the C3X and the car disconnected (USB-C), and then connect the C3X to the car after starting the car?

During this repetition, the TorqueControl learning values are initialized before learning is completed, so the process of continuous learning, initialization in the middle, and learning and initialization in the middle is repeated.

Which car does this affect?

Hyundai KONA EV 2ND GEN (HDA2)

Provide a route where the issue occurs

1618132d68afc876/0000000f--3a20636e0f

openpilot version

0.9.7 master-ci

Additional info

No response

mrkushaljoshi commented 5 months ago

I have a similar issue commaai/opendbc#1141. Vehicle Make and Model different however but the behavior is oddly similar.

sshane commented 5 months ago

@haram-KONA is this new?

haram-KONA commented 5 months ago

@haram-KONA is this new?

It keeps repeating little by little, but these days, the cycle occurs frequently.

haram-KONA commented 5 months ago

1618132d68afc876/00000011--e1f95ed2b7/0

This route occurred again this morning (KOR Time: Tuesday Apr 2 @ 09:45 - 09:45). “reboot” in this state, Engage will operate normally again. This happens over and over again, and the Trainning value of the torque-control is reset each time start driving, so it training again every time i start driving.

haram-KONA commented 5 months ago

Please save me~~~ The "mock" state keeps appearing and not appearing again and again, and the TorqueControl Training data disappears every day... so My Car just keep Training over and over again....

Please save me~~~

route : 1618132d68afc876/0000002a--865a3d5ef4

Verylukyguy commented 5 months ago

I have been having a similar unresolved issue https://github.com/commaai/opendbc/issues/1093 Every other start up results in Dashcam Mode and I need to restart the vehicle.

sshane commented 5 months ago

Until I have more time to check in depth, can you try this? https://github.com/commaai/openpilot/tree/simple-hkg-query

sshane commented 4 months ago

@haram-KONA please try that branch and report back.

haram-KONA commented 4 months ago

It is currently being used as " master-ci ", but it has not occurred yet since April 10th (KOR Time).

sshane commented 4 months ago

There may be a few bugs in Hyundai's VIN queries that were masked before this cleanup PR: https://github.com/commaai/openpilot/pull/32022

The cluster ECU we use to get the VIN on HDA1 cars for example can see our bus 0 queries on HDA2 cars, but is filtered out unless the multiplexing mode is set to the radar bus, not OBD port. Found by querying it on bus 0, then switching off OBD multiplexing: you see it on bus 1 and we don't respond to it - leaving it in a bad state.

Not sure if camera is related, but it very well could be. Checking it out on our EV6.

sshane commented 3 months ago

@haram-KONA I requested upload of some recent routes, can you put it on wifi to upload?

haram-KONA commented 3 months ago

@haram-KONA I requested upload of some recent routes, can you put it on wifi to upload?

@sshane now connected wifi - I am currently uploading logs from my drive 3 hours ago.

haram-KONA commented 3 months ago

@haram-KONA I requested upload of some recent routes, can you put it on wifi to upload?

@sshane " Hyundai CAN FD: match cluster speed commaai/opendbc#1110 " : Is this difficult to apply?

sshane commented 3 months ago

Please try to reproduce it by rebooting the device a few times, then you can request upload those logs

haram-KONA commented 3 months ago

Please try to reproduce it by rebooting the device a few times, then you can request upload those logs

all logs uploaded

sshane commented 3 months ago

We need logs from a dashcam route

sshane commented 3 months ago

So far we've discovered three things: on startup an unknown ECU requests DTC info from the camera (could be others as well):

    ( address = 1988,
      busTime = 0,
      dat = "\003\031\002\fUUUU",
      src = 1 ),
    ( address = 826,
--
    ( address = 1996,
      busTime = 0,
      dat = "\003Y\002\f\252\252\252\252",
      src = 1 ),
    ( address = 568,

and a few seconds later, it then requests the part number (0xF187). But this doesn't seem to prevent us from fingerprinting in most cases, as this is a bit after we should have fingerprinted.

The second discovery is that on this HDA2 car, you can query the cluster ECU on non-OBD bus 1 on bus 0 and it will send you a response on bus 1 that we ignore because we're specifically looking for a response on bus 0. This could be putting the camera into a bad state if this is the case as it waits for our flow control continue message that we never send.

Here only the last sent message returns a successful delivery status, but the ECU still didn't respond. If it didn't get the bus 0 query, it should have responded though which is odd. The only explanation is that it did actually send and put the ECU into a bad state, but the panda didn't report it or we dropped a CAN packet.

# OBD
sendcan: rxaddr=1988, bus=1, 2481.79 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

sendcan: rxaddr=1988, bus=0, 2595.65 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

# PT/non-OBD
sendcan: rxaddr=1988, bus=1, 2816.51 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

    can: rxaddr=1988, bus=129, 2818.93 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8
    can: rxaddr=1988, bus=129, 2818.93 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

The third discovery is sometimes our sent query doesn't return on bus 128/129 which could mean the camera never receives our message at all. I don't see any lost message increase in the few routes I've checked, so this could be an intermittent panda bug, possibly related switching to a disconnected comma power that racks up rx errors.

sshane commented 3 months ago

@adeebshihadeh ~~I do actually see a dropped CAN message right around the exact timestamp that we're missing a response which doesn't help, do you want to check this out? 1618132d68afc876/00000020--f400d529c7/0~~ this could be switching the ELM327 safety param

sshane commented 3 months ago

There's also some weird interaction with the OBD bus, it reports a response without a comma power! But no response if we don't query non-OBD right after, so there's some bug here. Will look into it

sendcan: rxaddr=1988, bus=1, 33983.36 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8                                                                                           

    can: rxaddr=1988, bus=129, 33989.22 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

    can: rxaddr=1996, bus=1, 34129.61 ms, 0x103362f100535832, b'\x103b\xf1\x00SX2', len(can.dat)=8                                                                                                       

sendcan: rxaddr=1988, bus=1, 34201.5 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8                                                                                            

    can: rxaddr=1988, bus=129, 34210.71 ms, 0x0322f10000000000, b'\x03"\xf1\x00\x00\x00\x00\x00', len(can.dat)=8

commaai / openpilot