Preliminary, not fully tested approach to building images for ExpressLRS bootloader

brad112358 commented 1 year ago

Olli, Please review this and let me know what you think of the approach I have taken here to build a second version of the R9M firmware with the proper flash address and small code changes to work with the ExpressLRS bootloader. This produces a .elrs file which can be flashed using OpenTX or ExperssTX without needing to erase the stock Frsky bootloader or solder wires for ST-Link.

I'm still working on testing this (a previous hand-built version worked well) and I haven't even started on user documentation, but I wanted to get your feedback before I continue this work.

Elsewhere you mentioned needing to move the isr table, but this is already taken care of by the ELRS bootloader since it sets the proper VTOR value to point to our isr table before jumping to our start address (which is extracted from our table as well).

Thanks!

olliw42 commented 1 year ago

VERY COOL!

just my 2 cents straight from my brain, not well thought out:

MLRS_USE_BOOTLOADER: pl rename to e.g. MLRS_FEATURE_ELRS_BOOTLOADER. MLRS_FEATURE_XXX are supposed to be the external/global define options. I would hesitate to introduce yet another notation, and to me it seems to fit the MLRS_FEATURE_XXX bag. maybe see also https://github.com/olliw42/mLRS/blob/main/mLRS/Common/hal/hal.h#L15-L34

in run_make_firmwares you should not test for the appendix, but for whether the flag MLRS_USE_BOOTLOADER (MLRS_FEATURE_ELRS_BOOTLOADER) is in the extra_D_list!

mlrs_use_bootloader: the sole reason seems to be that a respective symbol is defined, so one can use DEFINED(). It seems to me it's not actually relevant that it is set to 1, could be any value, right? Two other approaches come to mind

(i) creating a copy of the original ld script, maybe named STM32F103C8TX_FLASH_ELRS_BOOTLOADER.ld, with the changed origin and size. In run_make_firmware3 then the modified ld could be given to the linker. In some way I would like that better since it doesn't modify the orignal.

(ii) instead of declaring the symbol in the code, one could give it to the linker call by -Wl,--defsymb mlrs_use_bootloader (or however this has to be exactly, google should tell)

Btw, in order to handle the EE_SIZE 124 it would be cleaner if we would replace the tx-R9M-f103c8 target by a tx-R9M-f103cb target. Not very important momentarily.

I am curiously looking forward to how teh whole process turns out to be. :)

olliw42 commented 1 year ago

just for my understanding. I guess on thing which constantly confuses me is what the ELRS bootloader has to do with the stock Frsky bootloader. I.e.. how does it come that you can load the .elrs file with the stock Frsky bootloader.

Does it mean that the two bootloaders happen to be compatible? In the sense that you can have either in the flash of your system, and they will work just alike?
If so, are we then actually talking about two cases, case 1 = FrsKy bootloader is in flash and case 2 = ELRS bootloader is in flash?
If so, does this mean that if we wanted to use the .elrs all we would have to do is to flash the ELRS bootloader into our non-FrsKy mLRS board?

other question, wasn't it somehow necessary to modify the binary and do some encryption, or whatever it was? Where is this done? Would this be another script which needs to be called?

olliw42 commented 1 year ago

mlrs_use_bootloader: isn't there actually a 3rd possibility?

(iii) use preporcessor defines in linker script, i.e., something like

#ifdef MLRS_FEATURE_ELRS_BOOTLOADER
  FLASH    (rx)    : ORIGIN = 0x8004000,   LENGTH = 128K-0x4000
#else
 FLASH    (rx)    : ORIGIN = 0x8000000,   LENGTH = 64K
#endif

I found a google link which seesm to imply this. E.g. https://stackoverflow.com/questions/28837199/can-i-use-preprocessor-directives-in-ld-file

not sure it would be better than (i)

brad112358 commented 1 year ago

The linker doesn't use the preprocessor itself. So you can't use #ifdef in a linker script unless you explicitly run it through the pre-processor as a separate step before linking. The IDE doesn't do this (as far as I know), so we can't do it this way if we want the IDE to also work for builds.

olliw42 commented 1 year ago

in the IDE one could add it as a pre-build step, couldn't we?

brad112358 commented 1 year ago

just for my understanding. I guess on thing which constantly confuses me is what the ELRS bootloader has to do with the stock Frsky bootloader. I.e.. how does it come that you can load the .elrs file with the stock Frsky bootloader.

Yes, I suppose the OpenTX/EdgeTX and ELRS/Frsky bootloader thing is all a little confusing. I think this is, in part, because the ELRS folks seem not to want to talk about the Frsky firmware file format much.

As you know, using ST-Link to flash anything on fresh Frsky hardware requires clearing a lock which erases the entire flash including the stock Frsky bootloader.

The ELRS bootloader can be used with or without the stock Frsky bootloader so yes, we could put it on our non-Frsky boards and this would have some advantages such as allowing firmware update on those boards via OpenTX or EdgeTX which both have special support for the flashing protocol used by the ELRS bootloader. I think this firmware update method would be preferred over ST-Link by many mLRS users for the TX side even for non-FRsky TX.

The Frsky bootloader starts at the beginning of flash at 0x8000000. When used with the Frsky bootloader, the ELRS bootloader starts at 0x8002000 which is where the Frsky radio firmware normally starts. The ELRS bootloader installs ELRS (or mLRS) at 0x8004000. When the ELRS bootloader is installed by/along with the Frsky bootloader, it must be "signed" / "encrypted" via a secret. If the ELRS bootloader is installed via ST-Link, it resides at 0x8000000 and on Frsky hardware, would replace the stock bootloader if installed this way.

OpenTx and EdgeTx also have separate support for the proprietary protocol which the Frsky bootloader uses. This support can be used to flash images which are "signed" by Frsky as long as the stock Frsky bootloader is still intact. The "signing" process and secret was reverse-engineered so that ELRS could use the stock Frsky firmware via OpenTX or EdgeTX to install their own bootloader which, in turn can be used to install ELRS (or, now, mLRS).

The ELRS bootloader is handy for us because it allows us (with no extra development) to install mLRS via OpenTX or EdgeTX without needing to connect ST-Link. It also allows installing firmware via a serial port which can be used via the serial pass-through feature of Ardupilot or Betaflight to update the firmware on receivers without removing them from a build or using ST-Link.

Alternatively, we could design our own bootloader or we could add code to mLRS to support the open protocol which OpenTX and EdgeTX use to install ELRS firmware. But, the fact that a version of the ELRS firmware is already available and signed for Frsky saves us some work to produce our own signed bootloader, though, the secret has recently been made public by the developer who did the original reverse-engineering.

I am especially interested in supporting firmware install and update without ST-Link because I think it makes the R9M and R9mm easier to use for people with less than expert soldering skills.

brad112358 commented 1 year ago

in the IDE one could add it as a pre-build step, couldn't we?

I suppose, but I would prefer one of the other two methods. I chose the method I have used because it seems more self documenting to me and avoids having to have an entire second linker file. If you strongly prefer to have a second linker file, I'm OK with that.

You are correct that the initialization to 1 is not needed and, now that I think about it, is, perhaps, misleading. The overhead in the executable with the current method is very small, and if we leave out the initialization as you perhaps hinted, this should further reduce the footprint of this method.

brad112358 commented 1 year ago

MLRS_USE_BOOTLOADER: pl rename to e.g. MLRS_FEATURE_ELRS_BOOTLOADER

OK, that makes sense.

in run_make_firmwares you should not test for the appendix, but for whether the flag MLRS_USE_BOOTLOADER (MLRS_FEATURE_ELRS_BOOTLOADER) is in the extra_D_list!

OK.

Btw, in order to handle the EE_SIZE 124 it would be cleaner if we would replace the tx-R9M-f103c8 target by a tx-R9M-f103cb target. Not very important momentarily.

Hmm, yes, after you explained the (lack of) difference in these two parts, I wondered why you (anyone) wouldn't do that to begin with? I'm not sure what the process is to change this in the IDE at this point or if there would be any side-effects, so I'd like to leave this to you.

olliw42 commented 1 year ago

Yes, I suppose the OpenTX/EdgeTX and ELRS/Frsky bootloader thing is all a little confusing ... ...

ah, so we are actually talking of two kinds of ELRS bootloaders, one signed version which can be loaded using the Frsky bootloader, in which case we actually have two bootloaders in flash, and a normal version with which things work like it usually does with bootloaders.

THX for the clarifications. I'm sure you've explained all this to me before, but I just keep forgetting LOL

I think this firmware update method would be preferred over ST-Link by many mLRS users for the TX side even for non-FRsky TX.

well - except for R9 - I'm not so convinced it is so useful and worth the effort. I somehow have an impression that most ELRS users upgrade via the web interface, and if we would have a GUI/APP I think anyone would be happy with using the COM port for upgrading. So, I see potential here only for those mLRS txes which don't have a COM port. I find the STM32 system bootloader really ok.

Alternatively, we could design our own bootloader

I myself have zero interest in that.

The only bootloader I'm interested in is one on the receiver side, to allow OTA upgrade of the receivers.

Anyway, this was a bit OT, so back to topic :)

I would prefer one of the other two methods. I chose the method I have used because it seems more self documenting to me and avoids having to have an entire second linker file. If you strongly prefer to have a second linker file, I'm OK with that.

ok

I have a preference for the 2nd .ld method since it doesn't affect the code, and is clearer exactly as it does have the two linker scripts :). But I really do not have a strong feeeling, the current method while ingenious is a bit butt-backwards, and ackward overall. I mean, I can't see any practical advantage of the DEFINED() method. Is there any?

I think the footprint is no issue.

Since we always can change it later to what we later learn may be better, I would be ok with leaving it as is momentarily.

tx-R9M-f103c8 target vs tx-R9M-f103cb

in hindsight it is ackward it wasn't setup cb from the beginning. Anyway, it's really just for niceness, the only difference is that the flash size is different in the .ld script ... we always can also fake it there.

so I'd like to leave this to you.

you happily can do so. It was just a BTW. :)

summary

So, pl do the two other changes

MLRS_FEATURE_ELRS_BOOTLOADER
change in the py

Maybe also change mlrs_use_bootloader to

uint8_t or int (if uint8_t is not yet available). It's a tiny bit nicer to assign a value to an interger.
maybe rename it to something like mlrs_elrs_bootloader or maybe even mlrs_bootloader_x4000. I.e. something which would potentially allow us to use the mechanism also for other bootloaders.

Q1: is it actually possible to use the value of the symbol in the ld script, i.e., do something like ORIGIN = 0x800000 + mlrs_use_bootloader, LENGTH = 128K - mlrs_use_bootloader ?

Q2: The line ORIGIN = DEFINED(mlrs_use_bootloader) ? 0x8004000 : 0x8000000, LENGTH = 128K-0x4000 isn't totally correct but should be ORIGIN = DEFINED(mlrs_use_bootloader) ? 0x8004000 : 0x8000000, LENGTH = DEFINED(mlrs_use_bootloader) ? 128K-0x4000 : 128K, shouldn't it? Sure, the little loss of flash doesn't harm, but...

brad112358 commented 1 year ago

Should we provide a link in the documentation for people to download the bootloader binaries directly from the ELRS folks (It seems to be quite stable and not changed much these days) or do you want to add these images to your git repository? If so, where?

olliw42 commented 1 year ago

I don't have an opinion, i.e. would leave that to you to decide. I mean, the purpose of the exercise is to make it most easy for a user, so set it up in a way you think this is achieved. And at this point in time I really can't oveersee how it eventually will look like. All we possibly need to do is to properly cite any sources.

With respect to location, the mlrs-docs repo might be a location to host a complete guide including the tools, if that is reasonably possible. If not, well, just go ahead :)

brad112358 commented 1 year ago

Q1: is it actually possible to use the value of the symbol in the ld script, i.e., do something like ORIGIN = 0x800000 + mlrs_use_bootloader, LENGTH = 128K - mlrs_use_bootloader ?

Good question. Not that I've seen.

Q2: The line ORIGIN = DEFINED(mlrs_use_bootloader) ? 0x8004000 : 0x8000000, LENGTH = 128K-0x4000 isn't totally correct but should be ORIGIN = DEFINED(mlrs_use_bootloader) ? 0x8004000 : 0x8000000, LENGTH = DEFINED(mlrs_use_bootloader) ? 128K-0x4000 : 128K, shouldn't it? Sure, the little loss of flash doesn't harm, but...

Of course. It just didn't seem worth making the line any longer, but I'll add it.

olliw42 commented 1 year ago

thinking about the FrsKy/ELRS bootloader thing, I wonder: we actually would not need the ELRS bootloader, if we could/would create a frsky-signed firmware, right? Since the signing process seems to be known, now even publicly, why the heck then all the fuzz with the ELRS bootloader? Why not just a py script which creates this signed firmware? This should work also on the receiver side, shouldn't it? ???

brad112358 commented 1 year ago

thinking about the FrsKy/ELRS bootloader thing, I wonder: we actually would not need the ELRS bootloader, if we could/would create a frsky-signed firmware, right?

Perhaps, but I'd prefer to have as little dependency and involvement with the proprietary Frsky bootloader and format as possible.

Using the ELRS bootloader means a user can update via OpenTX or EdgeTX even if they have already used ST-Link and wiped out the Frsky bootloader. And, until we have a better method, installing via OpenTx or EdgeTX might also be useful for non-Frsky TX boards.

I think there are other advantages which the features of the ELRS bootloader can provide. As I mentioned earlier, I find it useful that it includes support for firmware updates over the serial port which I plan to use with the R9MM. At least in the short term since mLRS does not yet have over-the-air firmware updates of the receivers or an App to automate updates via a serial protocol. This is true even if the ELRS bootloader only ever gets used for Frsky hardware.

Also, perhaps most importantly, this is what I have working now and I don't know how much additional work might be required for a different approach. We don't have source for the Frsky bootloader which may make it more difficult to get mLRS to boot directly from it. The code changes to required to make mLRS work with the ELRS bootloader turned out to be very simple, but the ELRS bootloader source was very helpful while debugging early attempts at getting this working.

olliw42 commented 1 year ago

thx for the arguments as said, I will happily go with whatever you determine to be the best. I was just curious. :)

brad112358 commented 1 year ago

I have pushed an update which includes support for the small receivers. I have not added support for the receivers as TX or the Transmitter module as an RX as I'm guessing these are less popular. Let me know if you think if we should support the ELRS bootloader on these configurations too.

I have finished my testing and everything looks good. I was able to install and update mLRS on factory fresh R9M, R9MM and R9MX without the use of ST-Link.

I will work on the R9 documentation next.

olliw42 commented 1 year ago

very very cool!

I have not added support for the receivers as TX or the Transmitter module as an RX as I'm guessing these are less popular. Let me know if you think if we should support the ELRS bootloader on these configurations too.

yes, pl add them too. I think "all or none" :D

one minor suggestion inline

olliw42 commented 1 year ago

you came to a conclusion how you want to handle the elrs bootloader, i.e., from where to get it?

brad112358 commented 1 year ago

I plan to just provide links to the ExpressLRS images in the documentation.

geofrancis commented 1 year ago

I done some digging a while back and from what i could tell, I think Elrs have some frsky code, my understanding is that there is a frsky ELRS module so for it to be supported by frsky they had to make a special loader for it and thats where the ELRS method comes from. I suspect the binary is full of frsky source code thats why they dont want to talk about it.

brad112358 commented 1 year ago

From what I've seen, all of the ExpressLRS source code is available, including the bootloader, it's just the reverse-engineered tool to convert the bootloader to the proprietary Frsky format so it can be installed by the Frsky bootloader that they didn't publish.

brad112358 commented 1 year ago

I think this is ready to merge. I tested the R9MX as TX and R9M as RX only enough to verify they flashed OK and can connect to each other.

olliw42 commented 1 year ago

M A N Y T H X S I R !!!!!

geofrancis commented 1 year ago

From what I've seen, all of the ExpressLRS source code is available, including the bootloader, it's just the reverse-engineered tool to convert the bootloader to the proprietary Frsky format so it can be installed by the Frsky bootloader that they didn't publish.

I knew it was something to do with frsky code and where it came from but i didnt know the details.

olliw42 / mLRS

Preliminary, not fully tested approach to building images for ExpressLRS bootloader #104