search

adafruit / Adafruit_nRF52_Bootloader

USB-enabled bootloaders for the nRF52 BLE SoC chips
MIT License
445 stars 401 forks source link

Major upgrade to bootloader to decouple SoftDevice and self upgrade with uf2 #128

Closed hathach closed 4 years ago

hathach commented 4 years ago

This PR completely (mostly) decouple SoftDevice and Bootloader, however to be safe, and backward compatible the CDC interface can still be used to upload both bootloader + softdevice as it is previously.

Bootloader work with/without SoftDevice on the flash, it will detect the SD magic to correctly forward vector table to SoftDevice or nosd Application (such as circuitpython with other ble stack).

Extensive tests is performed as mentioned in #126 . A new bootloader Family ID is generated by uf2 online patcher with value of 0xd663823c. In addition, to prevent flashing the bootloader.uf2 from other board, the bootloader also check for BOOTLOADER_ID in cf2 (pinconfig.c) if available (which is a must now for all board). If it exists and not matches, the update will be aborted.

Note: also added Segger RTT printf for debugging.

Is your feature request related to a problem? Please describe. Decouple softdevice along with adding self uf2. There is a duplicate issue of some sort, but I create this new one anyway.

Describe the solution you'd like UF2 can be used to upgrade

  • Bootloader (family ID = 0xd663823c) , with check for BOOTLOADER ID in cf2
  • Application and/or SoftDevice (family ID = 0xADA52840)

For UF2 Softdevice can be bundled together with either bootloader or application into a single UF2 file.

CDC interface will left as it is as fail-safe since CDC can be used to revert to good-old version, should something bad happens.

Note: MBR is required to work with and without Softdevice (dynamically). MBR ROM & RAM requirement is as follows:

  • Flash 0x0000 - 0x1000 (first page)
  • RAM 0x20000000-0x20000008 ( first 8 bytes).

Check list

  • [x] uf2 works with or without a SD on the flash
  • [x] uf2 works with application only
  • [x] uf2 works with softdevice only
  • [x] uf2 works with application + softdevice
  • [x] uf2 works with bootloader only
  • [x] uf2 only update bootloader of the same board (using BOARD ID in cf2 config)
  • [x] uf2 work with CURRENT.UF2 read from device * [ ] uf2 works with bootloader + softdevice
  • [x] cdc works with application as it does now
  • [x] cdc works with bootloader + softdevice as it does now
  • [ ] ota should work when SD is on the flash: not tested, but should not change. This is subject to change in the next bootloader update

Also fix #24 #74

hathach commented 4 years ago

@dhalbert please test it out if you have time, there is lots of changes, though the most heavily changes is the ghostfat.c and msc_uf2.c . After this got merged, we can have it as beta for awhile. Though since CDC can be used to downgrade, it is rather safe.

jerryneedell commented 4 years ago

@hathach I tried this PR on a feather_nrf52840_express -- loaded via dfu-flash It loaded and seems to be working fine.

UF2 Bootloader 0.3.2-88-g1487239-dirty lib/intelhex (2.2.1-10-g5c15003) lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Express
Board-ID: nRF52840-Feather-revD
Date: May 12 2020

the "dirty" is because I had to modify the Makefile to use python3 on my system

jerryneedell commented 4 years ago

I noticed that the UF2-INFO.TXT does not report the SD version. Is that a change? I guess that may be the intent of this change ;-) It does on my feather_bluefruit_sense with bootloader 0.3.1 

UF2 Bootloader 0.3.1 lib/nrfx (v2.0.0) lib/tinyusb (legacy-1500-g23df777b) s140 6.1.1
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
Date: Mar  7 2020

after update

UF2 Bootloader 0.3.2-88-g1487239-dirty lib/intelhex (2.2.1-10-g5c15003) lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
Date: May 12 2020

works fine!

jerryneedell commented 4 years ago

FYI -- finally fixed my system -- via the magic of Python virtual environments, I can now build a "non-dirty" version

UF2 Bootloader 0.3.2-88-g1487239 lib/intelhex (2.2.1-10-g5c15003) lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
Date: May 12 2020

all good! Still curious if the loss of the SD version in the UF2-INFO was intentional.

hathach commented 4 years ago

@jerryneedell thanks for testing it out. The missing SD version in UF2-INFO.TXT is indeed intentional, since SD is not part of the bootloader anymore, the same bootloader (starting from this version), should be able to work with different Softdevice versions. It is application work now to show the bootloader version now.

However, if you think it is convenient to have it in bootloader mode, I can add a bit of detection code (for dyanmic SD) to have one extra line to INFO.TXT as well.

hathach commented 4 years ago

FYI -- finally fixed my system -- via the magic of Python virtual environments, I can now build a "non-dirty" version

UF2 Bootloader 0.3.2-88-g1487239 lib/intelhex (2.2.1-10-g5c15003) lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
Date: May 12 2020

all good! Still curious if the loss of the SD version in the UF2-INFO was intentional.

hmm, what is your trouble with the make ? I could try to fix it.

dhalbert commented 4 years ago

However, if you think it is convenient to have it in bootloader mode, I can add a bit of detection code (for dyanmic SD) to have one extra line to INFO.TXT as well.

I think that's a good idea, since it will help with support issues and make it clear what's already in flash. Something like "No SD present" vs "SD vx.x.x present", etc.

hathach commented 4 years ago

added with latest commit with an extra line for SoftDevice, since it is not part of the UF2 Version

UF2 Bootloader 0.3.2-89-g26e0248-dirty lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Express
Board-ID: nRF52840-Feather-revD
SoftDevice: not found
Date: May 13 2020

and 

UF2 Bootloader 0.3.2-89-g26e0248-dirty lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Express
Board-ID: nRF52840-Feather-revD
SoftDevice: S140 version 6.1.1
Date: May 13 2020
jerryneedell commented 4 years ago

@hathach Thanks for adding the SD version report.

the "issue" I had with the Makefile was taht in these lines:

$(BUILD)/$(OUT_FILE)-nosd.hex: $(BUILD)/$(OUT_FILE).hex
        @echo CR $(notdir $@)
        @python tools/hexmerge.py --overlap=replace -o $@ $< $(MBR_HEX)

# Bootolader only uf2
$(BUILD)/$(OUT_FILE)-nosd.uf2: $(BUILD)/$(OUT_FILE)-nosd.hex
        @echo CR $(notdir $@)
        python lib/uf2/utils/uf2conv.py -f 0xd663823c -c -o $@ $^

# merge bootloader and sd hex together
$(BUILD)/$(MERGED_FILE).hex: $(BUILD)/$(OUT_FILE).hex
        @echo CR $(notdir $@)
        @python tools/hexmerge.py -o $@ $< $(SD_HEX)

it invokes "python" On the system I am using now, I do not have Python2 installed so there is no "python" executable, only "python3" When I first ran it, I just changed "python" to "python3" and it built OK but makes it a "dirty" build. I now just uses a virtual environment with Python3 as "python" and it works fine. No changes necessary. So, I'm not sure it's a real "issue". I also ran it on a system that has Python2 installed as "python" and it builds ok

dhalbert commented 4 years ago

On the system I am using now, I do not have Python2 installed so there is no "python" executable, only "python3"

A user with a fresh install of Ubuntu 20.04 would also encounter this, so it would be good to explicitly use python3 here. Thanks.

jerryneedell commented 4 years ago

On the system I am using now, I do not have Python2 installed so there is no "python" executable, only "python3"

A user with a fresh install of Ubuntu 20.04 would also encounter this, so it would be good to explicitly use python3 here. Thanks.

That is exactly my situation!

hathach commented 4 years ago

I will change it to python3, just curious about windows. Does it name python3 as python3 or just simply python ?

Update: latest python installer have both python.exe and python3.exe. I will update it

jerryneedell commented 4 years ago

Built Bootloader for feather_nrf52840_sense without "python". No problems - Thanks for the fix.


UF2 Bootloader 0.3.2-91-g2df81e6 lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
SoftDevice: S140 version 6.1.1
Date: May 13 2020
hathach commented 4 years ago

@jerryneedell I think you should fetch prune the tag of tinyusb submodule, or maybe remove current and update init. The legacy tag of tinyusb is getting in the version numbering, that I removed recently.

jerryneedell commented 4 years ago

hmmm -- I did a fresh clone of the repo then fetched your PR

here is what I get

UF2 Bootloader 0.3.2-91-g2df81e6 lib/nrfx (v2.0.0) lib/tinyusb (legacy-2223-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
SoftDevice: S140 version 6.1.1
Date: May 13 2020

Here is what I did after cloning the repo.

F52_Bootloader$ git fetch origin pull/128/head:pr_128
From https://github.com/adafruit/Adafruit_nRF52_Bootloader
 * [new ref]         refs/pull/128/head -> pr_128
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/adafruit_github/Adafruit_nRF52_Bootloader$ git checkout pr_128
Switched to branch 'pr_128'
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/adafruit_github/Adafruit_nRF52_Bootloader$ git_update
Submodule 'lib/nrfx' (https://github.com/NordicSemiconductor/nrfx.git) registered for path 'lib/nrfx'
Submodule 'lib/tinyusb' (https://github.com/hathach/tinyusb.git) registered for path 'lib/tinyusb'
Submodule 'lib/uf2' (https://github.com/microsoft/uf2.git) registered for path 'lib/uf2'
Cloning into '/home/jerryneedell/projects/adafruit_github/Adafruit_nRF52_Bootloader/lib/nrfx'...
Cloning into '/home/jerryneedell/projects/adafruit_github/Adafruit_nRF52_Bootloader/lib/tinyusb'...
Cloning into '/home/jerryneedell/projects/adafruit_github/Adafruit_nRF52_Bootloader/lib/uf2'...
Submodule path 'lib/nrfx': checked out '7a4c9d946cf1801771fc180acdbf7b878f270093'
Submodule path 'lib/tinyusb': checked out '4e6aa0d81d2dbd23006d5a961938f6478049467a'
Submodule path 'lib/uf2': checked out 'adbb8c7260f938e810eb37f2287f8e1a055ff402'
Entering 'lib/nrfx'
Entering 'lib/tinyusb'
Entering 'lib/uf2'
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/adafruit_github/Adafruit_nRF52_Bootloader$ make BOARD=feather_nrf52840_sense clean
jerryneedell@jerryneedell-ubuntu-macmini:~/projects/adafruit_github/Adafruit_nRF52_Bootloader$ make BOARD=feather_nrf52840_sense

git_update does this

git submodule sync --recursive
git submodule update --init
git submodule foreach --recursive 'git fetch --tags'

am I missing a step?

hathach commented 4 years ago

@jerryneedell I am not entirely sure what is wrong, I only know enough git to use it. You should only see 2 release for the lib/tinyusb (without legacy one, which is removed in upstream already)

lib/tinyusb$ git tag -l
0.5.0
0.6.0

Btw, you don't have to clone submodules of tinyusb recursively, it will take lots of time due to the mcu driver and bootloader doesn't use that.

jerryneedell commented 4 years ago

@jerryneedell I am not entirely sure what is wrong, I only know enough git to use it. You should only see 2 release for the lib/tinyusb (without legacy one, which is removed in upstream already)

lib/tinyusb$ git tag -l
0.5.0
0.6.0

Btw, you don't have to clone submodules of tinyusb recursively, it will take lots of time due to the mcu driver and bootloader doesn't use that.

strange -- those are the only tags I see

lib/tinyusb$ git tag -l
0.5.0
0.6.0

also the commit for the tinyusb in my build is the same as yours -- just a different tag so am I getting the correct version?

lib/tinyusb (legacy-2223-g4e6aa0d8)

vs your build

lib/tinyusb (0.6.0-272-g4e6aa0d8)

No need for you to spend more time on this, I'll look into it and try it again after it gets merged.

jerryneedell commented 4 years ago

@hathach never-mind -- I am sorry to have wasted your time

My build is fine

UF2 Bootloader 0.3.2-91-g2df81e6 lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
SoftDevice: S140 version 6.1.1
Date: May 14 2020

I'm really no sure what was wrong before, but I think it was just a problem reading the INFO-UF2.TXT file properly -- I may have been seeing a cached version.

hathach commented 4 years ago

@hathach never-mind -- I am sorry to have wasted your time

My build is fine


UF2 Bootloader 0.3.2-91-g2df81e6 lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)

Model: Adafruit Feather nRF52840 Sense

Board-ID: nRF52840-Feather-Sense

SoftDevice: S140 version 6.1.1

Date: May 14 2020

I'm really no sure what was wrong before, but I think it was just a problem reading the INFO-UF2.TXT file properly -- I may have been seeing a cached version.

np at all, it is possibly cached by os. Did you test upgrading the bootloader itself by dropping the -nosd.uf2 file. Try to make some changes e.g increase version and rebuild

jerryneedell commented 4 years ago

@hathach never-mind -- I am sorry to have wasted your time My build is fine


UF2 Bootloader 0.3.2-91-g2df81e6 lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)

Model: Adafruit Feather nRF52840 Sense

Board-ID: nRF52840-Feather-Sense

SoftDevice: S140 version 6.1.1

Date: May 14 2020

I'm really no sure what was wrong before, but I think it was just a problem reading the INFO-UF2.TXT file properly -- I may have been seeing a cached version.

np at all, it is possibly cached by os. Did you test upgrading the bootloader itself by dropping the -nosd.uf2 file. Try to make some changes e.g increase version and rebuild

nice! that worked well -- i made a 0.3.21 version and copied over the -nosd.uf2 -- looks good and it did not erase the Flash!

UF2 Bootloader 0.3.21 lib/nrfx (v2.0.0) lib/tinyusb (0.6.0-272-g4e6aa0d8) lib/uf2 (heads/master)
Model: Adafruit Feather nRF52840 Sense
Board-ID: nRF52840-Feather-Sense
SoftDevice: S140 version 6.1.1
Date: May 14 2020
hathach commented 4 years ago

Yeah, it tries to use the highest address in the flash to avoid application, (below the internal fatfs) though there is still chance to corrupt the application if it is big enough. There will be another pr for that later. Right now we will try to have reliable way to upgrade both bootloader and app/sd first :)