mathias-arm
commented
1 year ago @polat-ahmet reported in #1032:
While flashing the max32666fthr board with Drag and Drop using max32625 debugger, it often fails(nearly 70% fail, %30 success). It's ok when I try with small size(~40kb) binary, but I'm having trouble with bigger size(~500kb) binary.
It was working properly before. I observed the problem using Windows updates KB5026361, KB5025221. When I uninstall these updates and tried it on KB5023696, I did not encounter any problems, successfully flash.
free2create
ozersa
microbit-carlos
fesc-q
top-5
selimgullulu
A recent Windows 10 and Windows 11 update has started triggering checksum and time our errors on DAPLink.
This has been reported by micro:bit & Calliope users, and we have been able to replicate in Windows 10 and 11 when the OS is kept up-to-date. We haven't tried Windows 8.1, but that has reached end of life last January.
Triggering Windows Update
The cumulative updates have been found and installed using this Microsoft catalogue: https://www.catalog.update.microsoft.com/Search.aspx?q=Cumulative%20Update%20Windows%2011%2022H2%20x64
Windows 11 22H2
I went through installing and uninstalling cumulative updates, and in my findings the problem is triggered when installing 2023-02 Cumulative Update Preview for Windows 11 Version 22H2 for x64-based Systems (KB5022913) from the 28th of February, which updates Windows 11 22H2 to OS Build 22621.1344.
The previous cumulative update KB5022845 from the 14th of Feb (OS Build 22621.1265) doesn't trigger this issue.
Windows 11 21H2
The Microsoft update catalog doesn't show any updates for Win 11 1 21H2 since November 2022, so I won't bother to test this Windows version.
Windows 10 22H2
The issue was triggered for me using 2023-03 Cumulative Update Preview for Windows 10 Version 22H2 for x64-based Systems (KB5023773) from the 21st of March, which updates the OS to build 1904x.2788.
The previous cumulative update KB5023696 from the 14th of March (OS Build 1904x.2728) doesn't trigger this issue.
Windows 10 21H2
We've also been able to replicate this issue in Win 10 21H2, and Microsoft is still releasing updates for this OS version, so it makes sense that we could identify a specific cumulative update to introduce this issue. I probably won't be looking into this one, nor Win 10 20H2 as it's unlikely to provide any additional useful information.
Failure modes
We've encountered a few different ways in which errors emerge:
The errors are not triggered on every flash, but different users have reported different error frequencies. In our internal testing some teammates measured 20% failure rate and others up to 60%. Some users have reported errors happenning on "almost every flash".
We've used micro:bit Universal Hex files for the majority of these tests, which are a bit more resilient to this issue (more info in the "Identifying the Cause" section), so other DAPLink users flashing Intel Hex files might encounter this problem more often (it's also likely that the micro:bit user that reported an error on "almost every flash" was using Intel Hex files as well).
Identifying the Cause
I’ve collected a couple of RTT logs from DAPLink with additional debug prints to track how the OS writes the file blocks to disk, and peaking at the actual data. While it’s still a bit early (I need more time to capture more data and analyse it), initial findings point at the problem being caused by file blocks being sent out of order by the OS.
In previous Windows versions, the file blocks are sent in order, but after the listed Windows updates are installed it looks like some file blocks are first sent as zeros, and then later down the file transfer the blocks are sent again with the real file data.
For example:
And this can happen more than once on the same file transfer.
However, not every file transfer sends files out of order, some are sent in order and it all works fine.
The check sum errors are encountered when the OS sends a block filled with zeros and DAPLink tries to calculate the checksum of an Intel Hex record. I still need to capture a better log for timeout errors, but I believe those are usually triggered when out of order blocks are ignored by DAPLink and then when the OS has finished sending the file, then DAPLink waits for more data to arrive (as the ignored blocks are not counted when measuring how much file data was transferred) until it eventually times out.
For the micro:bit specifically we use Universal Hex files, a superset of the Intel Hex format, which contains data for micro:bit V1 and micro:bit V2 in the same file. In file transfers where the out-of-order blocks correspond only for a section of the Universal Hex file that is not relevant the target MCU being flashed, the flash can still be successful. So while I haven't yet compared failure rates of Intel vs Universal Hex, it's very likely Intel Hex (and bin) files fail more frequently.
A checksum error log and Universal Hex file can be found here:
(Also note that because there is a lot of log data captured, data is sometimes dropped, so it might look like some blocks are not being sent, but we can look at the variables tracking the file size transferred to confirm that data has been processed, it's just that the RTT buffer was likely full).
Workarounds
Using
robocopywith the/zflag, for restartable mode, seems to be work so far.For example, with the terminal at the path where your
file.hexis located, and assuming DAPLink is mounted as driveE:\:Also, WebUSB flashing works, so for Intel Hex and bin files this demo from DAPJs can still flash the boards: https://armmbed.github.io/dapjs/examples/daplink-flash/web.html
For micro:bit Universal Hex files, with online WebUSB tool will work too: https://microbit.org/tools/webusb-hex-flashing/