Alternative bootloader?

[perillamint]

Quite out of scope of this project, but as all we know, default DFU bootloader is quite unstable. It does not work "well" on Mac or Linux.

I think alternative bootloader which supports standard DFU protocol can help firmware development in Linux or other non-Windows systems.

Could we build alternative bootloader which supports DFU protocol other then default buggy USB-MSD DFU?

[Ralim]

Hi, I'm completely open to building an alternate bootloader for the unit. I toyed with this before, however, most people seemed to prefer the MSD based bootloader for ease of use. Also considering the number of emails, messages and issues that are similar to #23 I don't overly want to advertise it too heavily.

It would certainly be possible to implement an alternate bootloader & then have a conversion firmware that swapped out the bootloaders. If you would find this useful though, I can start a GitHub project for this and try and get a replacement bootloader up and going.

[perillamint]

I think I could get some time to tinker with iron firmware 1-3 months later. (need to source another iron for SWD pin soldering, get enough time to spend hacking new bootloader, etc)

However I have one question before wiping out bootloader of my iron (or another TS-100): Could I get back into stock system when I flash "DO NOT FLASH_Flash Backup.hex" using ST-Link?

[Ralim]

That would be awesome!

That is exactly the purpose of that file, it's an entire read out of the entire flash of my iron when I bought it (original fw 2.12). For if someone did manage to wipe theirs :)

[Ralim]

So I'm thinking of forking dapboot and working to get that running on the iron. So that we can use web-dfu along with the normal dfu-util to update the iron.

Mostly going to need a fair bit of testing before I push anything too publicly.... Dont want to brick irons :)

[vifino]

I am also in favor of "normal" DFU, to be used with dfu-util. Much easier, IMO. Once mine arrives, I'll be happy to test. Got JTAG/SWD adapters and everything, so in case something goes seriously wrong, I'll be able to recover.

[Ralim]

the dapboot implementation supports both web based and the normal dfu-util tool :) Awesome, will be good to have another tester :)

[evyd13]

I'll be glad to have another bootloader as well - I'm in for testing! There's just one thing: How are we going to get to the pins? They are too small to solder without the risk of ruining everything.

[perillamint]

I purchased Olimex STM32-H103 board and flashed that "DO NOT FLASH_Flash Backup.hex". After some configuration (Pull down PC11), It boots successfully on that board.

I think I could start bootloader development on this board rather then directly on iron.

[Ralim]

Hey, Yeah I have been using one of the blue common f103 breakout boards to work on. I just haven't had much time. The only issue I could see with this is getting the bootloader working well without the crystal. As I'm not sure if it just works at room temperature, or if they are performing a calibration on the internal oscillator. Since I think the internal oscillator tolerance is wider than USB technically requires.

I am planning to fork dapboot for the iron as it supports the web dfu, and see if it can run without the crystal. (The ts100 does not have any crystals or timing references fitted)

[perillamint]

@Ralim I tested it using rcc_clock_setup_in_hsi_out_48mhz() instead of rcc_clock_setup_in_hse_8mhz_out_72mhz(). It enumerates well and works well.

Host used: Dell XPS 9550

Here is my fork of DAPBoot https://github.com/perillamint/dapboot

[Ralim]

Awesome, if you want to develop it in a branch of this repo I can give you push access if that's easier? (Lets us keep the code for the iron fairly central). Thoughts ?

[perillamint]

BTW, I tried to load original firmware using DAPBoot. It seems loads find but Stock firmware's USB does not enumerate with error -71

I suspect clock configuration but I'm not sure. Does anyone know any clue about this?

[perillamint]

Hey @Ralim, Could you make documentation about pads on small STM32F103 board in TS100?

I'm planning to get another TS100 in September and that information will be great help

[evyd13]

[vifino]

Wow! Looks like some progress happened!

My iron will be here in the next couple of days - tell me if you need testers.

[Ralim]

Hi all, @perillamint I'll try and document the pads for you. I recently picked up another one from work for working on the bootloader. I'm 99% sure it will be the clock configuration. The firmware expects the bootloader to setup the clocks for 72MHz. Also ensure the bootloader is compiling and fitting, as the firmware expects to start at 0x8004000 off memory (so therefore size of the bootloader should be < 0x4000).

The 4 pads on the bottom of the board are GND,VCC,SWDIO,SWCLK. In that picture off memory they are GND in bottom left, vcc in top left, and the other two are clk/dio (i just swapped jumpers till it worked).

Once we get the bootloader up and running we can make a conversion firmware that re-flashes it using the old bootloader.

[perillamint]

I dumped rcc memory regions and it differ quite a bit. I'll try document this changes and apply this hack on bootloader.

I think this change could not merged back into dapboot mainline, so I think only option is fork and maintain dapboot for TS100.

[Ralim]

Yeah, I agree that we will have to maintain our own fork of dapboot. Should not be too hard to maintain, and will just need to keep an eye on what happens to it.

[perillamint]

My backup iron arrived today. I soldered ST-Link and flashed my bootloader.

It successfully loads stock firmware (without USB) and OLED shows text on it. However when I put your firmware, it does not turn on display.

Do you have any idea about it?

[Ralim]

Interesting, I'm using the same initialisation code as they do for the clocks etc, does your code jump to 0x4000?

Not massively sure what would cause it to hang, I'm happy to try on my development iron shortly (hectic week at the moment).

Can I ask which firmware you loaded? Also if you can inspect the program counter on the crash. My current best guess would be that the bootloader is not placing the firmware at 0x4000 offset so when my code sets the nvic to that offset the interrupts get invalid addresses for the Irq handlers.

Off a very vague memory the bootloader uses 0x8000 as it's loading point instead. I probably could technically drop the nvic remap in my code as I'm fairly sure the official firmware does it as well. This would also make sense as I make heavy use of the interrupts where as their code was entirely software polling based except for the usb connection off memory.

[perillamint]

I succeeded! The problem was this: https://github.com/perillamint/dapboot/commit/9966fbd641f371386cdb7a19c8ccacbd7eb4daf1

TS100 stock firmware assumes its bootloader turned on some STM32 peripherals. After I turn on USB and I2C, it works just fine.

[perillamint]

Now, we need some way to flash patched DAPBoot bootloader using stock bootloader.

[perillamint]

BTW, DAPBoot is not quite "fool proof" -- it could brick itself when user put large blob on it. I think we should mitigate this problem.

[perillamint]

I fixed self-brick issue with https://github.com/perillamint/dapboot/commit/e3a5a633d44b422397f8cb4b01636647ab44d84a this commit.

IMHO, After developing user-friendly way to replace bootloader using stock bootloader(maybe some kind of bootloader updater firmware?), we could do some beta(or alpha?)-testing from that point.

What do you think @Ralim?

[Ralim]

Hi, Can you confirm if this bootloader protects you from trying to flash to anywhere below 0x8004000? I'm happy to make up some conversion firmware that would re-flash the bootloader. Just rather busy at the moment.

[perillamint]

@Ralim Since DAPBoot does not support DfuSe, I don't think user could put negative number on current_dfu_offset

https://github.com/devanlai/dapboot/blob/master/src/dfu.c#L232

However, if we need real fool-proof, I think I could extend this patch(https://github.com/perillamint/dapboot/commit/e3a5a633d44b422397f8cb4b01636647ab44d84a) little bit to implement only allows flashing command within this address range: (0x08000000 + APP_BASE) - (0x08000000 + FLASH_SIZE)

[perillamint]

BTW, I think I could work on bootloader flasher in next week (or 2-weeks later, depends on my other works :)).

Since stock bootloader is quite fragile, I think bootloader flasher should implement some safety mechanism like payload checksum check to protect devices from "hard brick" scenario.

Another question is: Should we "merge" DAPBoot fork into your firmware repository? or should we create GH group to maintain TS100 related repos? What do you think @Ralim?

[Ralim]

Good to hear, I wasn't sure of memory how much protection it had for trying to flash the wrong thing :) Mostly just asking :)

It might be worth adding the extra protection just in case, can't hurt to add it :)

Yeah, I'm hoping I'll have more time soon to keep working on the firmware for this, just been a bit too busy with working. The .HEX file itself has a rough checksum in it, and the dfuse protocol has checksums from memory. The biggest issue is not corruption in the file transfer but just getting the file there, which the dfuse should solve.

We cant really stop people flashing the wrong files, we just have to make it so that those diles cannot break the dfu bootloader :)

I'm thinking of making a Github org for community E-design products since I'm slowly working on a codebase for the ES120 as well. Then we could use that to organise all the future repositories? (Might not move this one for a bit since this link has been shared a fair bit). Thoughts?

[perillamint]

I think cloning this repo to new group and put migration notice on this repo will be enough for redirecting people to new repo.

BTW, I'm planning to buy ES120 next month. It looks quite nice. I guess this DAPBoot port will work on ES120 without huge effort. I hope they didn't lock flash on ES120 like TS100 did. :)

[perillamint]

Just FYI, I'm trying to mainline some of the patches from DAPBoot fork to original DAPBoot. Although we could not mainline all of our changesets, some parts like flash overrun protection will be helpful to other hardware.

https://github.com/devanlai/dapboot/issues

[Ralim]

Hi @perillamint I would have to transfer the repository rather than clone because I need to keep the links working, considering the number of people who seem to have issues finding the releases page I don't want to make them have to find a new home for it too :stuck_out_tongue: Also, would need to ensure all links transfer correctly and don't break.

Heh, the TS100 flash is not locked afaik (wasn't locked on any of my units?). The ES120 does not have the debug pins broken out, and also disables JTAG+SWD by default which makes it much harder. Also, all the wiring is done to generic gpio so have to use software bit-bang for all the comms which sucks.

Good idea to mainline the changes. Though the other option here would be to just have a build option for dapboot and not need to do different repositories entirely? (Thinking if all the changes basically end up mainline except for OLED support)

Edit: It might also just be easier to keep them separate atm and see about combining into a group if the ES120 firmware gets going & we want to do an alternate bootloader for it?

[EricJanVanDenBogaard]

Never had any problems with the boot loader on my Mac Book Pro, is it possible to revert to the factory firmware with a new/different boot loader ?

[Ralim]

You have been lucky :smile: It seems to be really hit and miss as to who has it work perfectly and who has issues.

The new DFU should completely support the factory firmware, they are separate pieces of code entirely. Eventually, we will also provide code to let you swap back as well just in case the newer factory firmware has any issues.

[thanks4opensource]

Just to clarify: Does flashing the Ralim (or other) firmware .hex file also flash/overwrite the DFU? I.e., if the .hex file is bad or something goes wrong with the flash can you end up with not only a non-working iron due to corrupted firmware, but a bricked one that can't be re-flashed successfully?

I looked at a binary file converted from 1.16.3 TS100.hex and didn't find the strings "DFU" or "3.43" anywhere in it. Those characters display on my TS100's LCD both before and after flashing so I assume the ascii is somewhere in its flash memory (unless it's maybe just a bitmap). That implies "no", but this wiki seems to say that flashing does overwrite the DFU.

If so -- and this is a naive suggestion which I'm sure you've already considered and rejected for technical reasons -- is there any way to specify the address ranges in the .hex to only flash the firmware and not the DFU? Are the address ranges page-aligned to the boundaries of the flash erase/write blocks? Is the DFU above or below the firmware in the memory map (is it even in the same address space?) or is interleaved in the middle?

Like I said, these are naive newby questions if someone has the time to answer them.

[vifino]

@thanks4opensource It does not, so it shouldn't be possible to completely brick it without flashing the bootloader seperately first. But even then, with a JTAG/SWD dongle, you should be able to recover, although that might take a while, depending on whether you are used to stuff like this, which I am guessing not.

The DFU is not able to overwrite itself, or at least shouldn't. Unsure of the default one having much protection, however. That'd be such a bad idea, unless it's first action is to copy itself into memory or something. Don't quote me on this. I am pretty sure it doesn't do that. If you read above, @perillamint says they are working on a bootloader flasher, which would be something you flash via the original DFU and just copies a payload into the bootloader region. Much safer-ish.

Regarding location, the bootloader is first, as far as I know.

[thanks4opensource]

Thanks for the education and allying my fears, @vifino

I actually am doing a bunch of ARM/STM development over ST-LINK/SWD right now, although I'm a newbie to it. I'd like to hack on the TS100 firmware myself but the idea of flashing a binary (hex) and hoping for the best scares me. I'm used to having OpenOCD/GDB/etc and I don't like the thought of debugging with, at best, printfs to the LCD display as the only view into what's going wrong. I've read the stuff above, but the idea of opening the iron and soldering SWD access onto the pads is even scarier than flash-bricking it. ;)

Wish there could have been a debug port. Maybe a jumper to turn the USB connector into SWD via a custom-wired connector. But it's an end-user product and no reason for MiniDSO (or whoever is the actual manufacturer/designer -- I'm still confused by the whole Chinese chain of Alibaba/Taobao/Amazon/Ebay) to do anything like that. It's a miracle they open-sourced the firmware. I'm grateful for that -- it's the whole reason I bought one in the first place.

[perillamint]

@thanks4opensource FYI: there is debug port. STM32 MCU board part has test pads on it and it is wired Vcc(3v3) GND SWCLK SWDAT. However, you must solder RESET pin on capacitor C17 since they're not broken out via test pads. Only that part (Soldering on C17) is tricky.

[perillamint]

@vifino Original DAPBoot has a bug that may brick itself when user puts large enough payload which could damage bootloader, but I fixed that bug and mainlined fix for that.

Except for that bug, there is no way (unless another bug discovered) to go back below 0x08004000, where bootloader sits on flash, so I think it could be considered as "safe for normal use"

[Ralim]

@perillamint You don't have to solder the reset pin if you use software reset on the SWD link to reset the mcu for programming :smile:. One less thing to solder :)

I think it would be entirely safe for use for all practical use cases. At least in this product, they gave us the debugging pins to connect to... "They" is miniware which is owned by e-Design as far as I understand the hierarchy.

[thanks4opensource]

Just want to say I understand about the debug test pads being brought out, and am very impressed that miniware/e-Design/whoever was so open with both the hardware and firmware. I voted for them with my money! ;)

When I said "Wish there could have been a debug port" I meant a physical jack/connector. Sorry I wasn't clear about that. And I understand why physical size constraints and target market goals make my wish impractical.

Very glad all of you were able to use the pads for development and give the rest of us your firmware/bootloader/etc. code. Thank you! As for me, at my current (low) skill level I'm not comfortable performing soldering brain surgery on my new, semi-expensive toy.

[vifino]

@perillamint Thanks for clarification, I already knew that you fixed it before, I just wondered about the default one, the one that gets shipped from the factory.

I have no doubts that your fork will be much better than the original DFU, safety-wise and usability-wise (at least for Linux/Mac users). I'll happily try to flash it, as soon as you got a flashing firmware that just writes dapboot into the bootloader region. Maybe not as the first one. 😄

[perillamint]

I think I could start working on bootloader flasher from Monday. I'm considering I should add USB CDC-ACM console for user interaction (check payloads SHA256sum, confirm flash using some crude Y/N prompt, show flashing progress, confirm it should wipe out itself to enforce next boot mode to bootloader, etc).

I want to hear comment about this. Will it helpful or not?

[thanks4opensource]

@perillamint -- Those are very good features.

[vifino]

@perillamint It definitly sounds very useful.

[Ralim]

@perillamint I wouldn't worry about the USB-CDC implementation, but would instead just print everything to the OLED and use the buttons for confirmation. That way the user doesn't need to bother trying to get the CDC port working just to be able to continue with the flash.

Verifying the checksum would be a good idea, and also verify it again once its been flashed as well.

You should be able to technically load the entire flasher into ram to speed up the process as well.

You can just set the magic key for entering into dapboot (RTC registers) rather than needing to wipe itself out :)

[perillamint]

@Ralim I considered using OLED for verification process but I worry about it is not suitable for displaying full SHA256sum of payload.

Maybe RTC is much more suitable for bootloader flag rather than erasing 0x08004000.

[Ralim]

You don't strictly need to show the entire hash, even if you just showed the last part then it would give a confidence boost. Or you could use a crc32 which will fit on the screen?

[EricJanVanDenBogaard]

only thing i noticed is on my macbook, when i "eject" the TS100 with the OS option, it boots again when you're not fast enough to pull the plug. But i don't see this as a problem when i have to use it by usb for updates. (i hope for a future USB-C version of the TS100, i think the non polarity plug feature a nice feature for that alone worth it)

[perillamint]

I started working on BL flasher. Currently, it "just" flashes bootloader, without prompting to user. After I finish porting OLED driver, it could show various information about bootloader payload and ask for confirmation before flashing bootloader.

http://github.com/perillamint/ts100-bl-flasher

[perillamint]

I successfully ported OLED driver. Now, I have to integrate button input and CRC32 generator before first "beta" release.

Also, I think my driver port could work in DAPBoot since they share same foundation(libopencm3)