Reception of old messages

[TorstenRobitzki]

Hello, I have an application where I use this wonderful library to be able to use a PEAK USB CAN adapter on my Mac. An embedded device is connected to the PEAK adapter. The application on the embedded device and the application on the Mac side worked flawlessly for some years now. Recently, I had to upgrade the PCBUSB-Library to version 0.12 (before that, I've used version 0.9), because the newer version of the Apples C++ compiler was requiring a native ARM library.

The application is a bootloader and installing a new firmware takes ~ 1 minute when using 1MBit CAN bus speed. Once in a while (1 out of 5-10), the protocol between Mac and embedded device is violated. The protocol stack then starts to report the reception of messages that were already received some time before.

To make sure, that the problem is not within the implementation of the bootloader protocol, I've attached a logic analyzer and indeed I was not able to confirm the double received messages.

When I see that problem, there are always multiple messages that where twice and they are then received in the same order as they where original received. Here is an example of a message type that is reported to be received by the Mac application:

...
CTL: 401: 06 00 47 10 00 00 01 00 8
CTL: 401: 06 00 48 10 00 00 01 00 8
CTL: 401: 06 00 49 10 00 00 01 00 8
CTL: 401: 06 00 4a 10 00 00 01 00 8
CTL: 401: 06 00 4b 10 00 00 01 00 8 <-- first time
CTL: 401: 06 00 4c 10 00 00 01 00 8
CTL: 401: 06 00 4d 10 00 00 01 00 8
CTL: 401: 06 00 4e 10 00 00 01 00 8
CTL: 401: 06 00 4f 10 00 00 01 00 8
CTL: 401: 06 00 50 10 00 00 01 00 8
CTL: 401: 06 00 51 10 00 00 01 00 8
CTL: 401: 06 00 52 10 00 7e 00 00 8
CTL: 401: 06 00 54 10 00 00 01 00 8
CTL: 401: 06 00 55 10 00 00 01 00 8
CTL: 401: 06 00 56 10 00 00 01 00 8
CTL: 401: 06 00 57 10 00 00 01 00 8
CTL: 401: 06 00 58 10 00 00 01 00 8
CTL: 401: 06 00 59 10 00 00 01 00 8
CTL: 401: 06 00 5a 10 00 00 01 00 8
CTL: 401: 06 00 5b 10 00 00 01 00 8
CTL: 401: 06 00 5c 10 00 00 01 00 8
CTL: 401: 06 00 5d 10 00 00 01 00 8
CTL: 401: 06 00 5e 10 00 00 01 00 8
CTL: 401: 06 00 5f 10 00 00 01 00 8
CTL: 401: 06 00 60 10 00 00 01 00 8
CTL: 401: 06 00 61 10 00 00 01 00 8
CTL: 401: 06 00 62 10 00 00 01 00 8
CTL: 401: 06 00 63 10 00 00 01 00 8
CTL: 401: 06 00 64 10 00 00 01 00 8
CTL: 401: 06 00 65 10 00 00 01 00 8
CTL: 401: 06 00 66 10 00 00 01 00 8
CTL: 401: 06 00 67 10 00 00 01 00 8
CTL: 401: 06 00 4b 10 00 00 01 00 8 <-- second time
CTL: 401: 06 00 4c 10 00 00 01 00 8
CTL: 401: 06 00 4d 10 00 00 01 00 8
CTL: 401: 06 00 4e 10 00 00 01 00 8
....

The message acknowledges the reception of a certain bit of data. The first byte (the 06) is an opcode, the next 4 bytes are an address, followed by a 16 byte length information. After the reception of the memory for the address 0x00016700, the received block is acknowledged and then, out of a sudden, the reception of the block at 0x00014b00 is received again.

I've traced such an error situation with a logic analyzer and can see, that the double messages are not send over the CAN bus. Beside the messages above, I suspect that there are the same amount of messages received twice. But as they contain only an opcode, I can not distinguish them from the ones that are expected at that moment.

To me, this looks like there is a reception ring buffer, what when either overflows or wraps is not empty anymore, but full and thus the application will receive the data twice.

[mac-can]

@TorstenRobitzki, can you tell me the hardware type (preferably the part number IEPH00...) and firmware version number of your CAN adapter, as well as the Mac hardware type and its OS version number. Your bootloader application runs with the highest possible bitrate. Did you take care of correct cabling and termination of the CAN bus? And did you pay attention to the system load (CPU and RAM) of your Mac while running the application?

I will try to reproduce the described scenario. This may take some time, as my release test suite includes a stress test that checks for number and order of received messages and did not show any errors last time.

[TorstenRobitzki]

@mac-can "unfortunately" I'm out of office this week. I could provide you with a serial number of the PEAK adapter I'm using (unfortunately I have only one, so I can not try an other one), as soon as I'm back.

The cabling is most likely not perfect, so I would expect to see some transmission errors once in a while. The bootloader protocol was developed with some transmission errors in mind. Blocks of the transmitted firmware are protected by CRCs and when an error occurs, blocks will be retransmitted.

I haven't tried to make this more reproducible by increasing the system load. But that's a good point, will try that once I'm back.

If you are interested, I could provide you the binary (or sources) of the bootloader client and an eval board with the bootloader running on it, if you want to try that.

Thank you for providing this wonderful library and providing it to the public!

[TorstenRobitzki]

Serial number is: IPEH-002022 22058

[TorstenRobitzki]

@mac-can I've done some tests again and tried to increase the load of my laptop by running a build in parallel. I do not see a largely increased frequency of the error happening (2 out of 11 failed).

But, I have made an other observation: If I do not reset the eval board after the occurrence of the error, the error will happen again and again. To make sure, that this is not some how triggered by an error condition on the eval board, I simply disconnected the PEAK adapter from the CAN bus and reconnected it again. After that reconnect, the error probability went back to less than 1.

Hope you can make any sense of this input.

[mac-can]

Serial number is: IPEH-002022 22058

Thanks, this is a PCAN-USB adapter (without FD), but with galvanic isolation. My PCAN-USB adapters are without this option. And it is an older one. As far as I know PEAK has recently redesigned its CAN adapter portfolio.

[mac-can]

@TorstenRobitzki Do you use Polling or Blocking Read for the reception on the Mac side?

[TorstenRobitzki]

@mac-can I'm using blocking reads:

...
fd_set fds;
FD_ZERO(&fds);
FD_SET(can_fd, &fds);

select( can_fd + 1, &fds, NULL, NULL, timeout_arg );

if (FD_ISSET(can_fd, &fds))
{
    TPCANMsg message;
    for ( TPCANStatus status = CAN_Read(can_handle, &message, NULL); status == PCAN_ERROR_OK;
        status = CAN_Read(can_handle, &message, NULL) )
    {
...

I could give you read access to the code if you find that useful.

[TorstenRobitzki]

@mac-can Taken your advice and updated the firmware of the PEAK adapter. I've run the scenario above for 30 times in a row, without running once into the problem above. I think that fixed it.

Thank you for advice and assistance

Torsten