hardbyte / python-can

The can package provides controller area network support for Python developers
https://python-can.readthedocs.io
GNU Lesser General Public License v3.0
1.27k stars 597 forks source link

Dropped CAN frames using Socketcan on Raspberry Pi #474

Closed Lauszus closed 5 years ago

Lauszus commented 5 years ago

I'm using this library for logging data sent on a CAN-Bus running at 500 kbit/s. The data is sent roughly at 1 kHz.

I have set up a CAN Notifier to receive the data in a thread and the data is then written to a queue in a listener:

class LoggerBufferedListener(can.Listener):
    def __init__(self, log_opened_event: Event):
        self._log_opened_event = log_opened_event
        self._buffer = Queue()
        self._lock = Lock()
        self._closed = False

    def set_closed(self, closed):
        with self._lock:
            self._closed = closed

    def is_closed(self):
        with self._lock:
            return self._closed

    def on_message_received(self, msg: can.Message):
        # Only put the CAN-Bus message into the buffer if it is a close/open command or we are currently logging data
        # and the arbitration ID matches any of the logger IDs
        if msg.arbitration_id == CANLogger.ID_CLOSE_OPEN or \
                (self._log_opened_event.is_set() and
                 msg.arbitration_id in [logger['id'] for logger in CANLogger.Loggers]):
            self.put_nowait(msg)

    def get(self):
        return self._buffer.get()

    def get_nowait(self):
        try:
            return self._buffer.get_nowait()
        except queue.Empty:
            return None

    def put(self, item, force=False):
        if not force:
            if self.is_closed():
                return  # Return if the reader is closed

        self._buffer.put(item)

    def put_nowait(self, item):
        if self.is_closed():
            return  # Return if the reader is closed

        try:
            self._buffer.put_nowait(item)
        except queue.Full:
            Logger.exception('LoggerBufferedListener: Buffer is full')

Another thread is doing nothing else than calling the get() method, the messages are then unpacked and converted to SI-units similar to how I did it in the viewer program:

https://github.com/hardbyte/python-can/blob/9f6f9188ff4ad78b0162925781d24625ca65ac50/can/viewer.py#L135-L159

The data is then written to RAM. After the logged is closed the data is then stored in an Influx database.

However it seems that messages are dropped somewhere, as the data often has gaps in it.

I'm using a MCP2515 and have enabled it using the following overlays in /boot/config.txt:

dtoverlay=mcp2515-can0,oscillator=16000000,interrupt=12
dtoverlay=spi0-hw-cs

My question is basically if anyone has any experience using Socketcan on a Raspberry Pi. Do you guys believe it's a hardware or software issue? I've considered using a different CAN tranceiver than the MCP2515 just to check if that is the culprit. If that does not work I might try to implement the logging in C instead, but would prefer to use Python, as that makes all the parsing of data and writing to the database much easier.

teebr commented 5 years ago

I gave it a try with a 1Mbps bus averaging around 4 frames / ms and had a lot of dropped messages. I didn't look into it too much (tried a few variants of threading/ multiprocessing, and reduced the code to just reading as frequently as possible - no data processing), I figured Python was just not able to clear the buffer quick enough.

Instead, I wrote a small C client which dumps the raw CAN data to a server, and did the processing on there (coincidentally I was also using Influx, so just used the same server that was running on)

christiansandberg commented 5 years ago

Have you tried candump?

Lauszus commented 5 years ago

@teebr thanks for the reply. I might give that a try. Is your code open source?

@christiansandberg no I have not.

teebr commented 5 years ago

@Lauszus It's not at the moment- it was a work thing and honestly it was very hacky, I'll see if I can make it into a gist though. In the meanwhile you can use the candump.c source as a basis for how to read from the socket(s).

I had two threads running: one polling for CAN messages and putting the frame and timestamp in a char array, and the other just used memcpy (with a mutex) to copy the buffer over and send it to a local server. The most lightweight way would just be to open a TCP socket with the server and continually dump bytes. Using libcurl to send the bytes over POST is maybe a bit more robust, but there's more overhead.

Lauszus commented 5 years ago

@teebr okay thanks! I'll give that a try.

TravisJoe commented 5 years ago

I have had good luck with spawning a CAN manager processes, to have a truly independent threads. Then using Queues to push outgoing messages and to receiving queue for incoming messages after they have been formatted and timestamped.

Lauszus commented 5 years ago

@TravisJoe I'm not sure what you mean? The Notifier class is launched in a thread and as you can see in my code above my listener does nothing else than just write the incoming messages in a queue.

TravisJoe commented 5 years ago

@Lauszus Without knowing all the code I am assuming you are using threads and not processes. Since in a python script is really just a single processes you still end up sharing resources when threading. Spawning a processes for only does your listener class would likely help.

Also hardware filtering is the best only way to reduce overhead, second best is filtering at the lower OS level sockets. Filtering in code is about as slow as you can get, especially with python creating new objects every time. Use the hardware or at least the lower level socket filters and avoid the "if" statement in a listener. If you have to filter do it outside of the listener after it is placed in a queue.

You can also use a dictionary like a C style switch statement which is better than multiple if statements when speed in python is needed.

You might also be able to increase the socket buffer size.

FabianInostroza commented 5 years ago

Hi @Lauszus, did you solve the problem?

I'm using an MCP25625 with an ARM board (slower than the RPi, only 400MHz) in a bus with about 1200 frames/s. I also notice there are some frames dropped and I found that python uses a lot of cpu (>60%) when the script being run connects to the can bus, even if the script does nothing with the frames.

I wrote a C program that uses BCM to downsample the frames and send them to a vcan. With that scheme the python script runs better in term of cpu usage (I haven't verified the frame drop, I was worried about cpu usage). This C program uses about 1% cpu.

Lauszus commented 5 years ago

@TravisJoe I've tried using processes as well with the same result. I do use hardware filtering using the can_filters argument. From what I have read the difference of using a dict instead of if statements is really neglectable.

Thanks, I'll try increased the socket buffer size!

@FabianInostroza thanks for the input. My problem is that I actually do not want to downsample it, as I am logging some data via CAN-Bus. It's starting to sound like the best way to solve this was just to write the logging code in C.

teebr commented 5 years ago

@Lauszus I fleshed out my original comment into a small package. You run Python stuff on a local server and a small C client on the Linux device to send/ receive on the CAN bus. There's a raw TCP connection between the two, so latency is low. CPU usage on a Raspberry Pi 3B is < 10% with 2000 messages / sec. Have a look, it might suit your needs or give you some ideas.

Lauszus commented 5 years ago

@teebr I just tested it and it works very well!

I have added pybind11 bindings to the client: https://github.com/teebr/socketsocketcan/pull/1. Let's move the discussion to your repo :)

thbuch commented 3 years ago

Hi, I am noticing the same frames missing when trying to log to blf. I tested with candump logging to log format and the same result. I tried to expand input buffer using: $echo 100048576 | sudo tee /proc/sys/net/core/rmem_max $echo 100048576 | sudo tee /proc/sys/net/core/rmem_default

Initial buffer was 1048576 but I still got missing messages.

Have you experienced that problem using a Raspberry Pi and the Seeed Studio 2CH CAN FD Hat? At lower data transfer speed it looks fine.

Lauszus commented 3 years ago

No I have not used that particular hat. FYI my repo is here in case you want to use it: https://github.com/Lauszus/socketsocketcan.

I've been using it for years now without any issues with dropped frames.

thbuch commented 3 years ago

Thanks Lauszus! If I understood it well, I need the Raspberry connected to CANbus with the c client and an extra Laptop receiving all this data with a python server. Is that right? However, I have to do all the job in the Raspberry. :(

What makes me wonder is that I have already tested candump which is already c and still have these missing frames. What hardware interface are you using?

Lauszus commented 3 years ago

No I actually run both the client and the server on the Raspberry Pi. The C++ code basically fills up a big buffer with all the frames. This means that the data might be delayed if the Python code can not keep up, but at least it does not drop any frames.

Check out this example: https://github.com/Lauszus/socketsocketcan/blob/master/examples/local_client.py

This is very useful when you want to log data and do not care about showing the data in real time.

If you want to show the data in some kind of UI, then you can set use_unordered_map to True and limit_recv_rate_hz to the frequency you want to update the data at. The C++ code will then simply override the frames based on the ID of the frame and only send the latest data to the Python code at the frequency given by limit_recv_rate_hz.

Refer to the source code for more information: https://github.com/Lauszus/socketsocketcan/blob/master/client/client.cpp.

I've not tested candump enough to tell if it drops frames as well.

I'm using a MCP2515 and a SN65HVD1040 CAN transceiver.

thbuch commented 3 years ago

That looks good! I'll give a try next week and come back with results (or more doubts). Thanks a lot!

thbuch commented 3 years ago

Hi, After some initial difficulties with the installation I managed to make some tests. I am having quite good results, and that has helped me to realize that I also have some precious problems. And still have.

As you can see in the attached image, I am having 'spi0.0 can RX-0: FIFO overflow' every certain time (not periodically) even with no programs running, just with the CAN adapter can0 UP. I am getting errors and overruns which coincide with these FIFO overflows and also a lot of dropped messages (I guess because I am doing nothing with this data)

Any ideas why is this happening? Thank you

Captura_FIFOOverflow

Captura_SPI_2

Lauszus commented 3 years ago

Hmm that is quite a bad ratio between packets and dropped frames.

FYI here's the output from two device using the older mcp2515 driver:

3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 10
    link/can 
    RX: bytes  packets  errors  dropped overrun mcast   
    2203286206 296825960 493     3138    493     0       
    TX: bytes  packets  errors  dropped carrier collsns 
    4386257    556986   0       0       0       0
2: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 10
    link/can 
    RX: bytes  packets  errors  dropped overrun mcast   
    602737646  92142407 365     3878    365     0       
    TX: bytes  packets  errors  dropped carrier collsns 
    10760786   1345116  0       0       0       0

At what frequency are you sending data?

thbuch commented 3 years ago

I am using a laptop with a Vector Canoe Application to simulate a Car's CAN Bus. I can count an average of 4 frames/ms. So, frequency is about 4KHz. Settings are 500 kBaud and statistics in Canoe show around 45% busload.

Lauszus commented 3 years ago

Okay. I can not replicate that in my setup. I have a busload of around 67 % when I log data and that is not causing any issues.

hartkopp commented 3 years ago

This seems to be a duplicate issue - and is probably done: https://github.com/hardbyte/python-can/issues/657#issuecomment-902634137

thbuch commented 3 years ago

Hi Lauszus, hartkopp! It's not yet done. Thanks both because your proposals are helping a lot but still having some issues. I need some more tests to discard my own errors and I'll bring back my results (and previous mistakes).

thbuch commented 3 years ago

Hi, I could improve quite a lot by uninstalling the old driver and setting correctly the can interfaces (thanks hartkopp) and setting big buffers (Reffer to issue #657). However, it only works fine for half an hour aproximately. Apparently, after this period, rx-buffer is full and I find gaps in the logging files compared to the one logged using Vector Canoe.

Is not clear for the difference between using big buffers in socket (issue 657) or using big buffers with the socketsocketcan. After certain time both buffers are overflown.

If I run in the terminal:dmesg | grep spi I am getting: mcp251xfd spi0.0 can0: RX-0: FIFO overflow.

Are you logging for long time? let's say for hours?

Thank you.

Lauszus commented 3 years ago

@thbuch no I do not log at 67 % bus load for hours only during short intervals. However I contentiously log when the bus load is 14 % and I do not see any gaps.

thbuch commented 3 years ago

Correct, when I use my device as you explain I got good results. Do you know any way to control the rx buffer? Check usage, clear it completely, etc...

Lauszus commented 3 years ago

No unfortunately not, as I've never had a reason to change the default size, so I have never looked into it.

thbuch commented 3 years ago

Ok. Thant reminds me that I tried BufferedReader and Notifier some time ago. Then I found the problems with the drivers. I will give another try...

hartkopp commented 3 years ago

Ok. Thant reminds me that I tried BufferedReader and Notifier some time ago. Then I found the problems with the drivers. I will give another try...

Did you run into these problems when just using candump von https://github.com/linux-can/can-utils ?

thbuch commented 3 years ago

I've repeated the test to be sure it has been done after "driver reparation". Unfortunately yes, I find these problems using candump. So, maybe is still a driver problem?

Here problems start after 10 minutes. This is not always like this. Yesterday's tests were fine for 30 minutes (2 channels) but errors when logging for 1hour.

I've done the test using: candump -l can0,0:0,#FFFFFFFF

So, all messages received at can0 are logged in a file. After 2h logging I have converted to ASC in order to compare it with the Canoe file using: log2asc -I candump-2021-09-10.log can0 -O Log.asc

I have checked dmesg | grep spi

[ 6.497600] spi_master spi0: will run message pump with realtime priority [ 6.509671] mcp251xfd spi0.1 can0: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 6.522363] mcp251xfd spi0.0 can1: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 6.555307] mcp251xfd spi0.0 rename5: renamed from can1 [ 6.604950] mcp251xfd spi0.1 can1: renamed from can0 [ 6.664473] mcp251xfd spi0.0 can0: renamed from rename5 [ 621.952753] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 628.696688] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 635.661376] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 661.748531] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 663.758923] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 685.650905] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 688.566302] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 701.666307] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 728.761803] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 795.960679] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 810.571652] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 834.665120] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 853.690851] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 876.269362] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 904.759268] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 906.470081] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 954.762014] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 956.472383] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 961.205520] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 995.571566] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 996.773511] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1059.466518] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1062.781606] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1085.268896] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1086.978275] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1088.583973] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1116.701352] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1127.268521] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1154.270692] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1156.681698] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1171.682989] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1173.286512] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1201.694450] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1201.694575] mcp251xfd spi0.0 can0: CRC read error at address 0x0010 (length=4, data=00 da 2b 4d, CRC=0x17a8) retrying. [ 1211.586145] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1212.788177] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1233.773886] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1259.681617] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1272.277048] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1287.281498] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1300.178256] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1340.380884] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1342.789047] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1374.218609] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1384.295790] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1396.592426] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1409.693455] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1414.215709] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1469.691062] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1483.996334] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1513.498144] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1596.006772] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1597.212055] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1610.102649] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1622.997604] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1647.993525] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1661.795521] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1678.510917] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1690.500764] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1703.501736] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1717.300484] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1719.011552] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1720.217021] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1730.502400] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1754.240811] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1809.616940] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1825.222859] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1828.241175] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1839.719484] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1859.242491] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1869.814165] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1900.520548] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1903.746381] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1938.806409] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1941.722800] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 1982.318520] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2001.248743] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2008.215621] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2039.325974] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2053.735500] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2067.018427] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2079.518450] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2081.629248] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2118.722025] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2134.320334] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2147.320758] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2172.312859] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2192.634254] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2201.619554] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2205.741092] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2225.255257] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2232.020469] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2233.226079] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2234.829857] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2247.636627] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2258.626954] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2273.632739] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2274.837317] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2298.324655] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2306.263827] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2327.229974] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2340.826638] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2342.536063] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2356.335242] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2362.775710] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2386.738643] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2414.329013] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2446.830380] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2479.542318] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2524.034547] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2540.050111] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2568.347643] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2582.548014] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2595.547820] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2611.336926] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2613.347626] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2629.245354] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2649.855490] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2662.641734] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2710.154884] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2726.359065] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2741.160430] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2779.850914] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2782.766978] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2799.286229] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2855.146425] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2856.350228] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2858.360335] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2868.845606] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2871.762463] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2873.365666] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2914.362741] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2954.166285] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2965.350138] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2967.061566] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2968.268067] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 2982.167706] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3010.068218] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3023.359495] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3036.755269] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3037.055495] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3038.260920] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3042.782050] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3051.661678] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3053.673150] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3056.796020] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3083.662206] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3089.298007] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3099.062584] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3100.270367] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3102.780144] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3131.176532] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3174.875760] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3188.677902] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3204.300320] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3227.369983] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3242.675486] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3247.807487] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3255.676570] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3266.671921] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3280.376356] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3286.311059] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3308.685168] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3319.776587] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3322.284715] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3335.790438] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3346.879566] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3388.381009] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3399.773392] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3407.313303] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3414.279118] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3414.578811] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3416.590563] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3453.575564] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3454.780976] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3457.591200] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3480.086988] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3523.818175] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3537.320449] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3544.383872] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3582.386134] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3584.096882] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3613.823830] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3623.298915] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3628.326361] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3641.826768] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3661.693054] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3685.287081] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3699.703206] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3713.313963] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3749.089923] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3752.105432] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3762.400847] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3782.696828] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3783.900717] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3820.203573] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3829.397914] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3840.805091] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3841.903667] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3842.710424] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3865.600917] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3867.711463] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3876.714095] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3888.316974] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3907.902289] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3909.612871] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3921.912603] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3948.305451] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3948.605338] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 3974.310179] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4000.412121] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4011.818757] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4021.308487] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4033.309211] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4053.805705] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4079.123222] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4100.823544] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4117.834099] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4126.709035] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4130.830797] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4141.408507] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4153.413935] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4158.349174] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4166.728052] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4180.850142] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4209.226896] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4212.348466] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4220.418537] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4261.419191] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4277.858059] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4308.122700] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4343.433441] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4372.730661] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4381.626279] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4382.831587] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4394.736785] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4427.341068] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4473.135292] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4474.340557] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4487.341798] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4529.638861] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4569.645325] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4570.851170] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4613.350225] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4682.355536] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4700.356505] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4710.651030] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4711.858882] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4713.376734] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4721.346977] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4726.376919] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4779.869673] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4786.340833] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4799.746333] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4810.440254] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4835.757517] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4845.946856] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4850.381719] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4882.759767] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4891.341231] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4894.760733] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4915.954923] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4963.888554] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4969.753665] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4970.958017] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4982.664155] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4983.869694] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4994.664290] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 4995.870278] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5005.154951] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5049.765001] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5060.358117] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5075.658064] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5076.866344] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5086.453319] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5100.875185] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5103.894108] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5112.875507] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5132.965263] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5162.396472] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5185.473610] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5187.898019] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5217.675069] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5218.880819] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5228.165617] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5229.370908] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5229.671399] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5265.167366] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5267.278390] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5300.379765] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5358.405816] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5364.372103] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5397.673510] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5398.881664] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5420.973408] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5456.980065] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5524.785332] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5555.772022] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5558.892497] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5569.887166] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5571.491110] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5579.779141] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5593.582860] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5628.900775] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5639.984077] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5641.694697] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5642.900754] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5655.921000] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5699.493474] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5715.423703] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5724.998170] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5736.498120] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5758.489594] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5814.703525] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5815.909279] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5860.420312] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5880.500929] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5889.996375] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5892.912988] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5945.706919] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5957.511019] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5994.711889] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 5995.917134] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6037.808893] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6048.815291] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6079.705920] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6100.805177] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6127.516517] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6197.519760] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6210.426966] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6244.813590] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6269.414622] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6296.817361] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6298.020758] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6313.431510] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6322.812733] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6324.818727] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6325.727408] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6356.454267] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6396.025846] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6407.821716] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6421.515378] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6423.226291] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6424.832034] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6427.957274] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6438.832658] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6441.956678] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6449.924214] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6476.924894] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6506.425517] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6506.725131] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6522.932465] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6535.927189] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6550.833118] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6579.829797] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6606.836051] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6618.426435] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6648.026590] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6676.238897] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6693.956154] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6703.934918] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6705.540320] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6719.741104] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6720.946623] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6724.471267] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6764.848538] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6777.849820] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6781.978781] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6826.437292] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6828.846108] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6830.049985] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6856.852383] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6925.963680] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 6952.962362] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7018.554435] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7030.450579] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7030.750948] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7043.044999] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7058.256690] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7103.052976] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7141.451467] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7170.262160] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7171.467767] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7211.768830] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7212.976530] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7238.870647] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7241.995558] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7278.066780] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7320.474294] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 7348.074439] mcp251xfd spi0.0 can0: RX-0: FIFO overflow.

hartkopp commented 3 years ago

dmesg | grep spi

[ 6.497600] spi_master spi0: will run message pump with realtime priority [ 6.509671] mcp251xfd spi0.1 can0: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 6.522363] mcp251xfd spi0.0 can1: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 6.555307] mcp251xfd spi0.0 rename5: renamed from can1 [ 6.604950] mcp251xfd spi0.1 can1: renamed from can0 [ 6.664473] mcp251xfd spi0.0 can0: renamed from rename5 [ 621.952753] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 628.696688] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. [ 635.661376] mcp251xfd spi0.0 can0: RX-0: FIFO overflow. (..) [ 1201.694575] mcp251xfd spi0.0 can0: CRC read error at address 0x0010 (length=4, data=00 da 2b 4d, CRC=0x17a8) retrying.

The FIFO overflows are somehow sporadic - but this doesn't make it better.

The number of dropped frames and detected errors displayed here https://github.com/hardbyte/python-can/issues/474#issuecomment-897680596 could be either a simple overload condition - or it could be caused by a slightly misconfigured CAN bit timing which might created errors on CAN controller level.

@marckleinebudde : Can you give a hint what kind of errors are counted in the device statistics?

marckleinebudde commented 3 years ago

rx_dropped if the driver fails to allocate an skb in the RX-path https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1521

rx_fifo_errors if the queue between IRQ handler and NAPI is full in the RX-path https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1528

rx_over_errors and rx_errors on RX-FIFO overflow https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1643

rx_fifo_errors if the queue between IRQ handler and NAPI is full and cannot deliver the RX-overflow error message https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1680

bus_error in case of bus errors https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1716

rx_errors in case of CRC bus error https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1732

rx_errors in case of Stuff bus error https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1740

rx_errors in case of Format bus error https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1748

tx_errors in case of NACK bus error https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1757

tx_errors in case of Bit1/Bit0 bus error https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1767 https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1775

rx_fifo_errors if the queue between IRQ handler and NAPI is full and cannot deliver the bus error message https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1785

rx_fifo_errors if the queue between IRQ handler and NAPI is full and cannot deliver the bus off error message https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1858

tx_aborted_errors, tx_errors in case of a TX MAB underflow message https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1964

rx_dropped in case of a RX MAB overflow message https://elixir.bootlin.com/linux/v5.14/source/drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c#L1985

thbuch commented 3 years ago

ok, I found the FIFO error I got at line 1661 (@marckleinebudde ). However, I don't really understand this code and no clue what should I do here. @hartkopp commented these errors may be overload condition. CPU overload? 50% can be considered overloaded? And about the CAN bit timing configuration. Is it related to sample-point? What are you exactly reffering to?

I also noticed that if I receive an ErrorFrame then I miss some messages from being logged. This is quite sporadic as well. (15-20 messages missed/ErrorFrame. In total i miss about 0.02% of the messages)

What can provoque I receive these ErrorFrames?

I am not really sure FIFO overruns and ErrorFrames are related.

hartkopp commented 3 years ago

I am using a laptop with a Vector Canoe Application to simulate a Car's CAN Bus. I can count an average of 4 frames/ms. So, frequency is about 4KHz. Settings are 500 kBaud and statistics in Canoe show around 45% busload.

4 frames per ms looks more like 78% busload on a 500kBit/s CAN.

Do you use CAN FD or only Classical CAN?

Can you check what canbusload can0@500000 tells you about the busload?

thbuch commented 3 years ago

Sorry, I should have updated this data. The part I was counting was 4 frames/ms but it was not all time the same. It is actually an average of 2 frames/ms or 2KHz if I check the total frames in a log. Canoe shows around 45% and canbusload can0@500000 around 48%

I use both CAN FD and Classical CAN. Classical CAN @500000 and CAN FD bitrate 500000 and dbitrate 2000000. I could check that if I set the configuration as CAN FD, I can set/get classical CAN frames too. However, as I had errors I preferred to make separate configurations. Unfortunatelly, I am still getting those ErrorFrames.

At the moment, until I solve the problem with ErrorFrames, I am focusing on classical CAN. I have all the setup and Simulation with Classical CAN.

thbuch commented 3 years ago

I am a little stuck. What can provoque I receive these ErrorFrames? I don't see the error frames in Canoe's logs.

hartkopp commented 3 years ago

Sorry, I should have updated this data. The part I was counting was 4 frames/ms but it was not all time the same. It is actually an average of 2 frames/ms or 2KHz if I check the total frames in a log. Canoe shows around 45% and canbusload can0@500000 around 48%

Makes sense. Even when the average load is relaxed you might have pile-ups that probably might lead to the input-FIFO overflow.

I use both CAN FD and Classical CAN. Classical CAN @500000 and CAN FD bitrate 500000 and dbitrate 2000000.

Ok. AFAIK the latest canbusload tool (from this repository) supports CAN FD too by providing a data bitrate.

I could check that if I set the configuration as CAN FD, I can set/get classical CAN frames too. However, as I had errors I preferred to make separate configurations. Unfortunatelly, I am still getting those ErrorFrames.

ErrorFrames usually show up when the CAN bus is not properly terminated or when the bitrate setting (especially both the sampling points) does not fit exactly. You might do some experiments here.

I remember having problems when having ONLY a CANoe instance on the other side, which disappeared when another CAN node was attached to the network that did some additional ACK'ing on the frames.

So tweaking the bitrate/samplingpoint settings is worth it ...

At the moment, until I solve the problem with ErrorFrames, I am focusing on classical CAN. I have all the setup and Simulation with Classical CAN.

Yep.

thbuch commented 3 years ago

Ok, thanks again. I'll come back with the results in some days.

thbuch commented 3 years ago

I tested setting different sample points and sjw and, to be honest, I could not see big differences. I changed recv(1.0) to recv(0.001) and it looks like I got much less Errors and FIFO overruns.

A curious test: I tested recording traces using a CanCase in a Laptop and the Seeed Can-Hat in a Raspberry. Both with the same python code except configuration of the interface. The log data from the laptop was fine, as expected. Raspberry pi's log data was missing some frames, even no errors or FIFO overruns. CPU load around 35-40% temp. 50ºC

I noticed small gaps of frames (not periodically) comparing the log files.

At the beginning I was thinking about a problem with the order of the frames, but I can actually see the gap in the timestamp. It looks like sometimes it stops receiving data. Any idea?

And very rarely I got CRC read error at address= 0x0010 (length=4, data=00 e9 c6 02, CRC=0x7bfa) retrying. I am not transmitting this ID. Is this an ErrorFrame generated by the CAN driver?

thbuch commented 2 years ago

Hi, Some news here! I recently had the oportunity of testing a PEAK_USBCAN_FD. In order to install the driver it I updated my Raspberry Pi 4 using: sudo apt-get update sudo apt-get upgrade

I have logged data during hours with the PEAK_USBCAN_FD and works perfectly for both CAN Classic and CAN FD. So, it looks the problem is not my code.

Then I tested back with the Seeed-Studio's 2Channel CANFD interface (dtoverlay=seeed-can-fd-v2) CAN Classic: · Logging with 1 channel active works perfectly. (it was not working before, maybe the update solved it?) · Logging with 2 channels simultaneously brings FIFO overruns and therefore data lost in the log file. CAN FD · Logging with 1 channel brings FIFO overruns and therefore data lost in the log file. · Logging with 2 channels not tested I tested many different bit timming combinations using PEAK Bit-Rate-Calculation tool: https://www.peak-system.com/Bit-Rate-Calculation-Tool.496.0.html?&L=1

Example: sudo ip link set can1 type can bitrate 500000 sample-point 0.75 dbitrate 2000000 dsample-point 0.75 restart-ms 10 berr-reporting on fd on

A relatively good range of sample points is 70-80%. Configurations out of this range make the system very inestable. Then I tried to change parameters such as TSEG1, TSEG2 and SJW The parameter I could not change is the bitrate Prescaler (default 1).

Some knows how can I change bitrate Prescaler value?

The difference with the Seeed-Studio interface with PEAK_USBCAN_FD is basically the clock frequency and communication with Raspberry Pi: Seeed Studio: 40MHz (SPI) and Peak 80MHz. (USB port)

Both are recognised as CANX using: ifconfig -a

That leads me to another question. Seeed Studio Can HAT installs CAN0 and CAN1 adapters. Then if I plug PEAK's is configured as CAN2. However if I reboot with PEAK's connected, then CAN0 is PEAK's and CAN1 is Seeed Studio's adapter.

How can I configure PEAK's always as CAN2 ?

Lauszus commented 1 year ago

@thbuch did you ever get the mcp251xfd driver working reliable? I'm now using 3 MCP2517FD in an application and I keep getting mcp251xfd spi0.1 can1: RX-0: MAB overflow detected. errors.

marckleinebudde commented 1 year ago

Hey @Lauszus, the mcp2517fd has a lot of errata, there's not much you can do about it, other than switching to the mcp2518fd. Are you using the latest kernel? In v5.17 the MAB overflow detected. error message is not printed anymore.

thbuch commented 1 year ago

Hi guys, Since I tested PEAK hardware I had zero problems and no need to make buffers, etc. Just using python-can library and their driver. So I change my project to use it. Thank you for your help.