Rare packet corruption

[caljay98]

Pretty sure this has something to do with the first and last block being written to, and maybe something to do with the generic single byte data being written and not being aligned to 32bits

[olale013]

This test was on a dev-board with jumper wires and sd breakout

Debug 1:

• 137,149 bytes written
• 12439 valid packets
• 2 invalid packets
• Case 1 (packet 43): least significant three bytes of timestamp written again after a valid timestamp
• Case 2 (packet 11607): missing the least significant byte of timestamp of 2 bytes of id or data

Debug 2:

• 141,022 bytes written
• 12786 valid packets
• 6 invalid packets
• Case 1 (packet 43): lower 3 bytes of timestamp duplicated
• Case 2 (packet 11607): missing lower byte of timestamp and 2 bytes of id or data
• Case 3 (packet 11793): lower 3 timestamp duplicated
• Case 4 (packet 11839): missing 5 bytes of id/data
• Case 5 (packet 12444): starts with 7e, then last byte of previous packet duplicated (packet counter did not increment)
• Case 6 (packet 12490): missing either id or upper two bytes of data

Debug 3:

• 135,567 bytes written
• 12287 valid packets
• 4 invalid packets
• Case 1 (packet 43): same issue as 43 above
• Case 2 (packet 11607): same issue
• Case 3 (packet 11742): first two bytes of timestamp duplicated
• Case 4 (packet 11788): missing 2 bytes of id/data

In all three cases I checked the buffer just before f_write to note the last packet. _The bad packets do not occur at the end of fwrite.

There's a weird pattern with the first two bad packets having the same packet count and the same issue. Other than that the issues seem to be random.

Average invalid packets for these three runs: ~0.03%

If it wasn't for the consistency in the first two bad packets I would call it a signal transmission error rather than code.

[olale013]

Testing with the dlm-sim...

Every 45-46 packets (of 11 bytes) there's a chance that a bad packet will appear. Usually, there's 45 consecutive good packets, sometimes 138, etc. but it's always a rough multiple of 45/46.

45*11 = 495 + some good bytes. Bad writes seem to happen at the edge of a sector (512 bytes) but not always. Forcing f_write to write exactly 512 bytes does not fix the issue.

UPDATES:

• Writing exactly 512 bytes, not syncing, and closing the file after a few writes still produces bad packets.
• f_write always thinks its writing the correct number of bytes based on the bytes written parameter.
• Made a demo to write "123456789" repeatedly. Wrote 322,299 bytes without a single error. Something about the interaction between f_write and the rest of our code is broken.
• Previous tests showed that bad packets don't happen at the end of a buffer dump, but they also don't seem to happen in the middle of a write. Doing one big 150kB write produced 0 bad packets.
• # of bad packets does not seem to be directly related to # of bytes written, # of writes, or # of swaps
• Bad packets never appear in memory
• Repeated writes with imperfect/large transfer sizes work. Adding swapping logic corrupts the writes but not memory.

[olale013]

Packets must be a multiple of 8 bytes long.

I still don't understand the underlying issue. f_write is perfectly capable of writing an odd-number of bytes.

The dlm-sim is now functional with a uniform packet size and a new append_packet function.

[olale013]

Spoke too soon. This fix appears to work for the dlm-sim, but not the dlm.

Using packet sizes of 16/24/32 bytes does appear to reduce the number of errors though.

[caljay98]

We should try sending some memory that is statically defined to see if the bug still happens and write that

[olale013]

Could be related:

CMSIS V2 appears to trigger a bug somewhere that causes memory corruption in the Ethernet code, particularly around the receive semaphore.