search

microbit-foundation / micropython-microbit-v2

Temporary home for MicroPython for micro:bit v2 as we stablise it before pushing upstream
MIT License
43 stars 24 forks source link

Create mechanism for determining filesystem layout for MicroBitMemoryMap and editors #23

Closed jaustin closed 3 years ago

jaustin commented 3 years ago

Replacing what was in the UICR for micro:bit V1, we need to have metadata for flash layout. This should live at the end of the last page of the image, and start (well, start when working backwards, so technically end....) with a magic so that the code in MicroBitMemoryMap can find the right location (eg https://github.com/microbit-foundation/codal-microbit/blob/master/source/MicroBitMemoryMap.cpp#L120 )

It is also required so that editors and CI pipelines can make structures use of auto-built images

dpgeorge commented 3 years ago

Implemented by b260810a69ce092ce6b5c289046b68ba59678b27. Comments at the top of src/addlayouttable.py are hopefully descriptive enough.

microbit-carlos commented 3 years ago

Thanks Damien!

So the current regions are:

SoftDevice    0x01000..0x1c000
MicroPython   0x1c000..0x61f24
Filesystem    0x6d000..0x73000

@jaustin @microbit-sam @martinwork are these what you would expect for MicroPython BLE partial flashing as well? (no need to do start implementation yet, we just need to prepare for the future).

The bootloader+scratch-pages starts at 0x73000, which is were the filesystem currently ends. Do you need another region for that, or is this good as it is?

martinwork commented 3 years ago

I think so, if I'm remembering right... Is 61f24 = FLASH_PROGRAM_END?

1c000 and 73000 are the start and end of the app region (end of softdevice and start of flash scratch). MicroBitMemoryMap starts looking for magic in the next page after FLASH_PROGRAM_END (62000?). The start address of the region to partial flash is stored with the magic and hashes.

dpgeorge commented 3 years ago

Note that the MicroPython implementation of this flash layout is not compatible with the existing MicroBitMemoryMap, because the magic and format of the table is different.

martinwork commented 3 years ago

Where is documentation for MicroPython magic and table format?

microbit-carlos commented 3 years ago

Where is documentation for MicroPython magic and table format?

In the Python script at the moment: https://github.com/microbit-foundation/micropython-microbit-v2/commit/b260810a69ce092ce6b5c289046b68ba59678b27

microbit-carlos commented 3 years ago

@martinwork did you have a chance to continue looking at the region table from the iOS app perspective?

We had a look with @microbit-sam from the Android partial flashing side, and it looks good. The magic and table format is a different, so we still need to implement that in the CODAL MicroBitMemoryMap first, but just want to make sure the current format (things like the gap between the MicroPython and Filesystem regions) will be fine for the iOS app as well?

martinwork commented 3 years ago

I think it'll be OK. Just to check I understand...

It looks like mp_end is what MicroBitConfig calls FLASH_PROGRAM_END. Its the next byte after the end of the application code and data sections: https://github.com/microbit-foundation/micropython-microbit-v2/commit/b260810a69ce092ce6b5c289046b68ba59678b27#diff-17dc2ae0471db7ff8730ea93be18b772a1216a2621476fea78977cae6b101063R203

So with C113, the table will have regions: Softdevice: 0x1000, 0x1B000 MicroPython: 1C000 , mp_end - 1C000 FileSystem: fs_start, fs_end - fs_start

The table is at the end of the last page containing mp code and data, if there's room, otherwise at the end of the next page.

Will a partial flash will be just the filesystem? Will fs-start always be on a page boundary? Does the app need to know if the filesystem sizes and positions match?

What range would a full flash send? From mp_start to fs_end?

martinwork commented 3 years ago

Is the space from mp_end to fs_start unused apart from the table?

dpgeorge commented 3 years ago

Will fs-start always be on a page boundary?

Yes.

Is the space from mp_end to fs_start unused apart from the table?

Yes.

microbit-carlos commented 3 years ago

Will a partial flash will be just the filesystem?

Yes, is the MicroPython region changes we should go for a full flash.

Does the app need to know if the filesystem sizes and positions match?

Between the flash in the micro:bit and the hex file to be flashed? Probably not, but if the check is made they should match. If the MicroPython region hash matches, then we can assume the regions table should match and should have the same start/end addresses. Our current view is that it should erase all the filesystem pages and then flash whatever is in the hex file for that region. The hex file will not contain records with empty data (0xFF) so, if all the fs pages aren't erase, then left over files from previous programmes could stay there after a partial flash.

What range would a full flash send? From mp_start to fs_end?

Not sure about this one, the editor will provide a full Universal Hex (or could pass an Intel Hex for a given target). @microbit-sam can you answer this one?

microbit-sam commented 3 years ago

What range would a full flash send? From mp_start to fs_end?

Yeah that looks right

MicroPython 0x1c000..0x61f24 Filesystem 0x6d000..0x73000

Edit: in the Android app I currently extract up to 0x77000 when creating the update hex I think we could do up to 0x73000 and it'd leave the CODAL storage and scratch pages intact

microbit-carlos commented 3 years ago

The main disadvantage of only flashing up to 0x73000 is that things like the calibration data get erased on drag&drop and WebUSB flashing, but not on BLE full flashing (I assume this also means it doesn't get erased in BLE partial flashing?).

Where is the BLE bond data stored? is it expected to be preserved between full-flashes?

martinwork commented 3 years ago

I think MakeCode WebUSB does a partial flash whenever it can, which doesn't wipe calibration data.

The V2 BLE bond data is in FDS: 0x75000 and 0x76000. From memory, V1 bond data is in the same page as the calibration data. It always has been preserved by BLE flashes, unless something goes badly wrong!

microbit-carlos commented 3 years ago

It always has been preserved by BLE flashes, unless something goes badly wrong!

Yeah that makes sense, I guess in that aspect MakeCode partial flashing has always diverted vs drag&drop flashing.

On the other hand Python Editor WebUSB partial flashing does erase the compass calibration data, so it behaves the same as drag&drop, and this is still true for V2.

jaustin commented 3 years ago

This isn't a MicroPython issue and I'm happy with the discrepancy for now, but we can pick up as part of Python Editor discussion if we see it confusing people (probably us...)