Occaisional corruption of the downloaded hex file --> The hex file cannot be decoded. Checksum calculation failure occurred

jaustin commented 7 years ago

This is a tricky to track down issue, so hopefully posting it here we'll find other people with the same problem...

Though we haven't ever seen or reproduced it ourselves in the foundation, a number of people have sent us hex files that are missing large chunks of data, and the result is that the checksums fail and DAPLink refuses (correctly!) to flash the file.

The error DAPLink puts in the fail.txt is: The hex file cannot be decoded. Checksum calculation failure occurred

Missing chunks are not on line boundaries.

ntoll commented 7 years ago

An example of several of these hex files would be very useful!

jaustin commented 7 years ago

Yes, we've got to get permission to share them, which is why I haven't yet uploaded one - as I said, we can't reproduce ourselves yet :/

ntoll commented 7 years ago

Ugh... I presume the permission is needed because this is the work of someone under 16?

jaustin commented 7 years ago

Well, in one case yes but irrespective of age, we don't share people's hex files in a public forum (and hence their code, could contain all sorts of things!) without first asking permission :)

On 12 Jul 2017 7:20 pm, "Nicholas Tollervey" notifications@github.com wrote:

Ugh... I presume the permission is needed because this is the work of someone under 16?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bbcmicrobit/PythonEditor/issues/55#issuecomment-314854434, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI-qV_gGrf2m2uBfeiVYRrO8ftBw7mlks5sNQ54gaJpZM4OWAKs .

jaustin commented 7 years ago

We've got permission to share the following files - one is a corrupted hex and the other is the result of 'File-->Save as' on the page that was giving bad data.

The user believes there's no user visible virus software or browser security extensions and plugins.

confusing broken editor files.zip

ZanderBrown commented 7 years ago

Only contains one hex file, the other is just the .html of the editor.

Am I missing something or is the user attempting to download the hex via CTRL-S and saving the webpage rather than a hex generated by the webpage?

ntoll commented 7 years ago

So, I can confirm it's something at the user's end of things. I did a 3-way diff to tell this:

Corrupt hex is corrupt as reported by the user. (A)
Source of the editor containing the MicroPython runtime in a div contains exactly the same hex content apart from the user's script which is in the right place. (B) This means the editor did its job and inserted the user's script into the right location in the hex.
The content of the two previous hex (A and B) are different to the hex in the published editor (C). I can confirm that C is working.

Put simply, the version of the editor on the user's local machine has a hex file embedded into the HTML that is corrupt.

The 3-way diff demonstrated that large chunks of the hex served as part of the editor (C) are missing and replaced by lines of the wrong length (i.e. they're not the expected uniform length before a newline).

While the user might not think there's anything interfering with their communications, the above proves conclusively that there is. The source code delivered by python.microbit.org is correct, but what they see is different, ergo, something is interfering.

Do they have a proxy? Have they manipulated the HTML in any way? Are they or their ISP using some sort of filter that manipulates the contents of HTTP responses? In any case, the problem is firmly in their realm. Perhaps clearing the caches and reloading might do the trick?

One way we could mitigate these problems is by checksumming the content of the hidden div and checking it in client-side JS when the user tries to make a hex file.

As for the user's next steps, depending on how technical they are, I'd advise them to check their network stack to ensure there's not a proxy, filter or some other thing interfering with things. This may be at the ISP level.

Other than that, there's nothing we can do if we're serving correct content. :-(

jaustin commented 7 years ago

Could we please leave this open and tagged 'needsmoreinfo' or something to suggest we don't believe it's explicitly a problem with the code itself? That way there's something visible for other users seeing the same problem to find it? I think we don't really understand it yet and if someone were to show up on GitHub with the same issue and some tools to help debug it might be really useful.

Your conclusion is the same as ours here, by the way - something somewhere is corrupting the HTML file and as a result the hex file.

One thing @microbit-matt and @carlosperate noticed was that Chrome 'chunks' the long string in the inspector view, maybe a possible target for investigation.

carlosperate commented 7 years ago

Did we end up getting the users' chrome version? This could be important to identify the problem, in case it's a browser bug. Maybe this can be of some help as well: https://bugs.chromium.org/p/chromium/issues/detail?id=161127

ntoll commented 7 years ago

Reopened as per @jaustin's request.

ntoll commented 7 years ago

My biggest concern is that we're trying to fix a problem with a user's system over which we have no control.

The content is served correctly, via TCP/IP (so we can presume transit via the network isn't corrupting it). That leaves everything within the realm of the user. The best we can possibly do is simply flag that we can't do anything but point the user to check their own software/stack.

If it's a bug in Chrome, then it should be reported to Google in such a way that it's easy to recreate the problem condition. But is that the responsibility of the micro:bit foundation or the user who has the buggy browser?

Could they try it in another browser to confirm it's Chrome specific?

carlosperate commented 7 years ago

A few people have reported the same problem, and taking in consideration only a small % of people would actually make the effort to report it, imagine how many have just moved on and used something else. So perhaps we can mitigate it if we understand the problem a little bit better.

One thing we did wonder before is if the DOM was manipulated by any browser plug-in? Maybe a firewall or Ad-blocker. It's probably not the case, but if suddenly this didn't work for all uBlock users, then perhaps it's something we should address, so understanding why it's happening might still be worthwhile.

What you suggested earlier could be a way of mitigating it, perhaps fetching the HEX as an independent file and doing some type of checksum to ensure it is valid.

One disadvantage of doing this could be that we could lose the ability to allow users to just download a github zip and open the html file. If we definitely want to maintain this feature, we can still implement it in a way where we only do a XMLHttpRequest as a fallback (i.e., store the hex as a javascript string in a file, check the contents, and if needed then fetch again via XMLHttpRequest).

fabgui commented 6 years ago

Hello,

This post is fairly old but I might have a corrupted file for you.

I am very new to MicroBit but fairly computer competent (rapsbery pi, linux, python, ++). The kids brought two MicroBits back from school and played with them for a while. This evening, my daughter called after being unsuccesfull a downloading her latest hex file (enclosed). Windows was showing an 0x800703EE error and would not allow download of the file. We tried:

changing batteries
changing battery pack
downloading known-working HEX from the microbit website, called "first experience" and "smiley". Both failed to upload to the MicroBit
downloading HEX file with the "Microsoft micro:bit uploader". No apparent error message but it did not bring the Microbit to life.
upgrading to latest firmware. The firmware upgrade seem to have worked: new version in DETAILS.TXT but I was still unable to upload firmware.
Clearing out the Microbit using: https://support.microbit.org/support/solutions/articles/19000053246-clearing-out-the-previous-program-on-a-micro-bit. But there doesnot seem to be any link to the two required files; does any of you have the mentioned auto_on.cfg and erase.cfg files ?
we even tried that same suspicious HEX on a 2nd working Microbit ... it now seems bricked as well.

Here are the symptoms:

Yellow LED flashes a couple of times then stop (on the firmware-upgraded Microbit); yellow flashes then remain solid (on the 2nd Microbit with no firmware update, version 0241 enclosed)
Both Microbit are detected by Windows and appears as drive D:
Able to browse the content D: (DETAILS.TXT, MICROBIT.HTM, etc)
Fails whenever an HEX download is attempted.

This suspicious HEX was made from the MICROBIT online 'block' interface on Chrome 63.0.3239.84 with no ad-blocker and only the 'default' add-ons, antivirus sis BitDefender free 10.21.1109, basic windows firewall. That very same setup had issued operational HEX just a few minutes before this suspicious HEX. The initial download of the suspicious HEX was made under Microbit Firmware Interface 0241. They had both the USB and 3V connected... not sure if they are supposed to do it this way but they claimed that's how they do it at school and had done it that way numerous time at home before.

I now have two 'bricked' Microbit. Both were previously successfully updated using that very same setup. I don't think it is some power failure during transfer or ESD zap or ... It really seems to be a corrupted HEX.

Do any of you have any tips on how to recover them ? If more info would be useful, please shoot.

[removed.hex.zip]() While we work out what's going on I've removed the hex in question. Though I can't reproduce the failure on the setup I have right now, if this hex does cause trouble for other users I'd like to limit exposure to it until further investigation is complete. I'm very happy to share it with other developers or debuggers -- jaustin. ASSERT.TXT DETAILS.TXT

jaustin commented 6 years ago

@fabgui thanks so much for finding this ticket and reporting the issue here. The three files are very useful. As this issue tracker here deals specifically with the editor at python.microbit.org I'm not going to re-open this particular one, but please don't consider that as me not taking the issue seriously - as we draw together a range of open source projects for micro:bit we have to make sure we're filing issues where the right people will see them.

As this may be an issue with MakeCode, with the DAPLink firmware on the micro:bit, with the micro:bit DAL, or something about your setup, so could you please file a ticket at support.microbit.org? We can then work with you to diagnose the issue, and @microbit-mark and I will link this issue to that ticket so we can come back and update the status here for anyone in the future who finds this via search when we know a bit more about what's going on.

In the meantime, in the interests of some further debugging, the next thing I'd try is using one of the micro:bits on a different machine and flashing any other hex file to the micro:bit. Please only do it with one of the devices, though, because if that works, then debugging the other one to find out what has happened would be very useful!

Finally, It looks like @microbit-mark already has an updated draft of the knowledge base article that includes attachments for the erase and auto_rst files, so we can hopefully get that one fixed ASAP.

fabgui commented 6 years ago

Thanks for the prompt reply.

I quickly tried flashing a known-good HEX from a different computer (it might have been with a different cable ... not sure). It worked: I was able to recover one of the two Micro:bits (the one with original firmware). As you suggested, I'll keep the other MicroBit as is.

Later tonight, I'll try: 1) flash a known-good HEX from the original PC 2) flash the suspicious HEX from the original PC -- see if it fails again 3) flash the suspicious HEX form the 2nd PC

Anything else, you'd like me to try ? The above will help narrow down if it is the PC or the HEX file; once I know more I'll open a ticket on support.microbit.org.

fabgui commented 6 years ago

Here is the outcome of the follow-up testing:

flash a known-good HEX from the original PC --> works
flash the suspicious HEX from the original PC --> failed again (ie the suspicious HEX crashed the microbit again)
flash the suspicious HEX from the 2nd PC --> works

Based on this, it appears to be the combination PC + HEX that makes the microbit crash ... puzzling.

jaustin commented 6 years ago

Thanks very much for the extra debugging.

Looking at the assert you received, and the DAPLink source, it seems there have been some changes to the assertion behaviour in the v0244 version of the firmware:

Here's the 'assert' file for 0243 https://github.com/ARMmbed/DAPLink/blob/0243/source/usb/msc/usbd_msc.c#L1081

And here it is in 0244 https://github.com/ARMmbed/DAPLink/blob/0244/source/usb/msc/usbd_msc.c#L1085

The change suggests a bug was fixed, and so I'd like to see if 0244 still causes the issue for you. If it does, we will file this as a bug against DAPLink.

Could you please try this with the attached firmware build? (Ideally first check it still works on the 'dodgy' PC, then check whether it can still be broken on the 'dodgy' one).

Ideally, please keep the second micro:bit in it's soft-bricked state so we can still investigate that one.

0244_kl26z_microbit_0x8000.hex.zip (it'll need unzipping before using)

fabgui commented 6 years ago

Thanks. I tried with firmware v0244 on the 'dodgy' PC:

firmware uploaded OK
known-working HEX upload fine
suspicious HEX fails to upload and 'crashes' the microbit
then, moved to the 2nd PC and the suspicious HEX uploads fine.

This behaviour seems quite systematic regardless of whether the 3V battery pack plugged in or not during HEX upload.

--> same behaviour on v0244 as seen on v0243 and v0241.

microbit-carlos commented 6 years ago

Thank you for the additional @fabgui. Could you also upload the new DETAILS.TXT file from your micro:bit with the 0244 firmware?

Also, it would really help us if you could open a ticket in the support system (https://support.microbit.org), that way we could better track this internally, rather than continue the conversation in this issue, as it was originally intended to cover a different topic. Once we figure out what might be the origin of this issue we can open a ticket on the relevant repository (and link here in case somebody else encounters the same issue and finds this page).

fabgui commented 6 years ago

Here is the DETAILS.TXT with v0244. I'll make a ticket on support.microbit.org tomorrow. DETAILS.TXT

fabgui commented 6 years ago

Ticket #4174 opened to track this issue on support.microbit.org .

llsc12 commented 5 years ago

hey! i managed to get that error every time! here is my program, tried it on 2 different microbits and same result: The hex file cannot be decoded. Checksum calculation failure occurred.

error.hex.zip

llsc12 commented 5 years ago

this is a previous file i had that works fine, tried that and it works but the latest code dont work microbit-time-4-school!.hex.zip

microbit-carlos commented 5 years ago

Hi @llsc12,

Thanks for the report. These look like hex files created with the MakeCode editor, and this issue tracker is for the Python editor. Are you sure you've attached the right hex files? For any issues on MakeCode you can open a ticket in this repository: https://github.com/Microsoft/pxt-microbit/issues

llsc12 commented 5 years ago

sos i didnt know

On Mon, 28 Jan 2019 at 14:36, Carlos Pereira Atencio < notifications@github.com> wrote:

Hi @llsc12 https://github.com/llsc12,

Thanks for the report. These look like hex files created with the MakeCode editor, and this issue tracker is for the Python editor. Are you sure you've attached the right hex files? For any issues on MakeCode you can open a ticket in this repository: https://github.com/Microsoft/pxt-microbit/issues

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bbcmicrobit/PythonEditor/issues/55#issuecomment-458155636, or mute the thread https://github.com/notifications/unsubscribe-auth/AoxG3S2Y5TD4Q5tYsptCh0nhDCMT6djfks5vHwrugaJpZM4OWAKs .

microbit-carlos commented 5 years ago

No worries @llsc12, we are all happy to help. If those are MakeCode hex files, could you open a new issue on the MakeCode tracker? https://github.com/Microsoft/pxt-microbit/issues We will be able to help you over there a lot better than here.

petulino777 commented 1 year ago

It seems, github make some processing on microbits hex files. After download from GitHub - error message form Makecode Web: "Sorry, we could not recognize this file". Perhaps Github is not for binary files. It is enough to make a binary file compare between origin HEX file and uploaded/download HEX file. There are many repositories, all with invalid HEX files for micro-bits and the owners are happy. I made now this FC /B f1.hex f2.hex and they are not the same! ;-(

PPPDUD commented 1 year ago

I'm experiencing this issue while reflashing the firmware of a micro:bit v2.00 on Windows 11. I've tried both current and beta builds of the firmware for v2.00, but I always get the checksum error.

Edit: Bootloader version 0255

martinwork commented 1 year ago

@PPPDUD I suspect the problem you are seeing is different to the one in this old issue.

Please see this article. https://support.microbit.org/support/solutions/articles/19000145576-error-transferring-file-to-the-micro-bit-since-windows-update-in-february-2023

Until the Windows problem is fixed, dragging a hex file to the MICROBIT drive may require several tries.

bbcmicrobit / PythonEditor

Occaisional corruption of the downloaded hex file --> The hex file cannot be decoded. Checksum calculation failure occurred #55