MCUdude / MegaCoreX

An Arduino hardware package for ATmega4809, ATmega4808, ATmega3209, ATmega3208, ATmega1609, ATmega1608, ATmega809 and ATmega808
GNU Lesser General Public License v2.1
247 stars 52 forks source link

Optiboot and Flash library #116

Closed MCUdude closed 3 years ago

MCUdude commented 3 years ago

First of all, I've used some code I found over at the official Optiboot repo. Using this code I've been able to create a working optiboot.h/cpp library that my new Flash library wraps around. For reference, Optiboot.h/cpp contains the low(ish) flash read-write functions, while Flash.h/cpp contains high-level code that lets you write anything to the flash memory, like strings, structs, and variables. Both libraries will most likely work without modification on the tinyAVR-0/1/2 series too, so feel free to steel this @SpenceKonde!

@technoblogy I was not able to get the Read_write_without_buffer example working the megaAVR-0 series though. Do you have any experience with the "modern" AVRs and "write-to-flash" functionality?

optiboot.h implements optiboot_page_erase, optiboot_page_filland optiboot_page_write (borrowed from here), but I'm not sure these works differently than on classic AVRS? Here's the example sketch I've not been able to do what I want:

Code:

```c++ // This is just because my dev board has the USB to serial adapter connected to UART3 (default pins) #define Serial Serial3 #include // Allocate one flash pages for storing data #define NUMBER_OF_PAGES 2 const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__((aligned(SPM_PAGESIZE))) PROGMEM = {}; // Function for writing 16-bit integers to a flash page void flash_write_int(const uint8_t base_addr[], uint16_t offset_addr, int16_t data) { // Start by erasing the flash page before we start writing to the buffer if ((offset_addr & (SPM_PAGESIZE - 1)) == 0) optiboot_page_erase((uint16_t)base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); // Write the 16-bit value to the buffer optiboot_page_fill((uint16_t)base_addr + offset_addr, data); // Write the buffer to flash when the buffer is full if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2)) optiboot_page_write((uint16_t)base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } // Function to write bytes to a flash page void flash_write_byte(const uint8_t base_addr[], uint16_t offset_addr, uint8_t data) { static uint8_t low_byte = 0; if ((offset_addr & (SPM_PAGESIZE - 1)) == 0) optiboot_page_erase((uint16_t)base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); if (!(offset_addr & 0x01)) // Address is 2, 4, 6 etc. low_byte = data; else // Address is 1, 3, 5 etc. optiboot_page_fill((uint16_t)base_addr + offset_addr, (data << 8) | low_byte); if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2)) optiboot_page_write((uint16_t)base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } // Function to force a flash page write operation void flash_end_write(const uint8_t base_addr[], uint16_t offset_addr) { // Write the buffer to flash if there are any contents in the buffer if ((offset_addr & 0xFF) != 0x00) optiboot_page_write((uint16_t)base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } void setup() { delay(2000); Serial.begin(9600); static uint16_t addr = 0; Serial.print(F("Filling up flash page 0 with 16-bit values...\n")); // Fill the first flash page (page 0) with 16-bit values (0x100 to 0x01FF) for(uint8_t data = 0; data < (SPM_PAGESIZE / 2); data++) { flash_write_int(flashSpace, addr, data + 0x0100); // Write data addr += 2; // Increase memory address by two since we're writing 16-bit values } // Force an end write in case it hasn't already been done in flash_write_int flash_end_write(flashSpace, --addr); Serial.print(F("Filling up flash page 1 with 8-bit values...\n")); // Fill the second flash page (page 1) with 0-bit values (0x00 to 0x0FF) for(uint16_t data = 0; data < SPM_PAGESIZE; data++) { addr++; // Increase memory address by one since we're writing 8-bit values flash_write_byte(flashSpace, addr, data); // Write data } // Force an end write in case it hasn't already been done in flash_write_byte flash_end_write(flashSpace, addr); Serial.print(F("Flash pages filled. Reading back their content.\nPage 0:\n")); for(uint16_t i = 0; i < SPM_PAGESIZE; i += 2) Serial.printf(F("Flash mem addr: 0x%04x, content: 0x%04x\n"), flashSpace + i, (pgm_read_byte(flashSpace + i + 1) << 8) + (pgm_read_byte(flashSpace + i))); Serial.println(F("\n\nPage 1:")); for(uint16_t i = 0; i < SPM_PAGESIZE; i++) Serial.printf(F("Flash mem addr: 0x%04x, content: 0x%02x\n"), flashSpace + SPM_PAGESIZE + i, pgm_read_byte(flashSpace + SPM_PAGESIZE + i)); } void loop() { } ```

Output:

``` Filling up flash page 0 with 16-bit values... Filling up flash page 1 with 8-bit values... Flash pages filled. Reading back their content. Page 0: Flash mem addr: 0x0400, content: 0xff00 Flash mem addr: 0x0402, content: 0xff01 Flash mem addr: 0x0404, content: 0xff00 Flash mem addr: 0x0406, content: 0xff01 Flash mem addr: 0x0408, content: 0xff00 Flash mem addr: 0x040a, content: 0xff01 Flash mem addr: 0x040c, content: 0xff00 Flash mem addr: 0x040e, content: 0xff01 Flash mem addr: 0x0410, content: 0xff00 Flash mem addr: 0x0412, content: 0xff01 Flash mem addr: 0x0414, content: 0xff00 Flash mem addr: 0x0416, content: 0xff01 Flash mem addr: 0x0418, content: 0xff00 Flash mem addr: 0x041a, content: 0xff01 Flash mem addr: 0x041c, content: 0xff00 Flash mem addr: 0x041e, content: 0xff01 Flash mem addr: 0x0420, content: 0xff00 Flash mem addr: 0x0422, content: 0xff01 Flash mem addr: 0x0424, content: 0xff00 Flash mem addr: 0x0426, content: 0xff01 Flash mem addr: 0x0428, content: 0xff00 Flash mem addr: 0x042a, content: 0xff01 Flash mem addr: 0x042c, content: 0xff00 Flash mem addr: 0x042e, content: 0xff01 Flash mem addr: 0x0430, content: 0xff00 Flash mem addr: 0x0432, content: 0xff01 Flash mem addr: 0x0434, content: 0xff00 Flash mem addr: 0x0436, content: 0xff01 Flash mem addr: 0x0438, content: 0xff00 Flash mem addr: 0x043a, content: 0xff01 Flash mem addr: 0x043c, content: 0xff00 Flash mem addr: 0x043e, content: 0xff01 Flash mem addr: 0x0440, content: 0xff00 Flash mem addr: 0x0442, content: 0xff01 Flash mem addr: 0x0444, content: 0xff00 Flash mem addr: 0x0446, content: 0xff01 Flash mem addr: 0x0448, content: 0xff00 Flash mem addr: 0x044a, content: 0xff01 Flash mem addr: 0x044c, content: 0xff00 Flash mem addr: 0x044e, content: 0xff01 Flash mem addr: 0x0450, content: 0xff00 Flash mem addr: 0x0452, content: 0xff01 Flash mem addr: 0x0454, content: 0xff00 Flash mem addr: 0x0456, content: 0xff01 Flash mem addr: 0x0458, content: 0xff00 Flash mem addr: 0x045a, content: 0xff01 Flash mem addr: 0x045c, content: 0xff00 Flash mem addr: 0x045e, content: 0xff01 Flash mem addr: 0x0460, content: 0xff00 Flash mem addr: 0x0462, content: 0xff01 Flash mem addr: 0x0464, content: 0xff00 Flash mem addr: 0x0466, content: 0xff01 Flash mem addr: 0x0468, content: 0xff00 Flash mem addr: 0x046a, content: 0xff01 Flash mem addr: 0x046c, content: 0xff00 Flash mem addr: 0x046e, content: 0xff01 Flash mem addr: 0x0470, content: 0xff00 Flash mem addr: 0x0472, content: 0xff01 Flash mem addr: 0x0474, content: 0xff00 Flash mem addr: 0x0476, content: 0xff01 Flash mem addr: 0x0478, content: 0xff00 Flash mem addr: 0x047a, content: 0xff01 Flash mem addr: 0x047c, content: 0xff00 Flash mem addr: 0x047e, content: 0xff01 Page 1: Flash mem addr: 0x0480, content: 0xff Flash mem addr: 0x0481, content: 0x00 Flash mem addr: 0x0482, content: 0xff Flash mem addr: 0x0483, content: 0x02 Flash mem addr: 0x0484, content: 0xff Flash mem addr: 0x0485, content: 0x04 Flash mem addr: 0x0486, content: 0xff Flash mem addr: 0x0487, content: 0x06 Flash mem addr: 0x0488, content: 0xff Flash mem addr: 0x0489, content: 0x08 Flash mem addr: 0x048a, content: 0xff Flash mem addr: 0x048b, content: 0x0a Flash mem addr: 0x048c, content: 0xff Flash mem addr: 0x048d, content: 0x0c Flash mem addr: 0x048e, content: 0xff Flash mem addr: 0x048f, content: 0x0e Flash mem addr: 0x0490, content: 0xff Flash mem addr: 0x0491, content: 0x10 Flash mem addr: 0x0492, content: 0xff Flash mem addr: 0x0493, content: 0x12 Flash mem addr: 0x0494, content: 0xff Flash mem addr: 0x0495, content: 0x14 Flash mem addr: 0x0496, content: 0xff Flash mem addr: 0x0497, content: 0x16 Flash mem addr: 0x0498, content: 0xff Flash mem addr: 0x0499, content: 0x18 Flash mem addr: 0x049a, content: 0xff Flash mem addr: 0x049b, content: 0x1a Flash mem addr: 0x049c, content: 0xff Flash mem addr: 0x049d, content: 0x1c Flash mem addr: 0x049e, content: 0xff Flash mem addr: 0x049f, content: 0x1e Flash mem addr: 0x04a0, content: 0xff Flash mem addr: 0x04a1, content: 0x20 Flash mem addr: 0x04a2, content: 0xff Flash mem addr: 0x04a3, content: 0x22 Flash mem addr: 0x04a4, content: 0xff Flash mem addr: 0x04a5, content: 0x24 Flash mem addr: 0x04a6, content: 0xff Flash mem addr: 0x04a7, content: 0x26 Flash mem addr: 0x04a8, content: 0xff Flash mem addr: 0x04a9, content: 0x28 Flash mem addr: 0x04aa, content: 0xff Flash mem addr: 0x04ab, content: 0x2a Flash mem addr: 0x04ac, content: 0xff Flash mem addr: 0x04ad, content: 0x2c Flash mem addr: 0x04ae, content: 0xff Flash mem addr: 0x04af, content: 0x2e Flash mem addr: 0x04b0, content: 0xff Flash mem addr: 0x04b1, content: 0x30 Flash mem addr: 0x04b2, content: 0xff Flash mem addr: 0x04b3, content: 0x32 Flash mem addr: 0x04b4, content: 0xff Flash mem addr: 0x04b5, content: 0x34 Flash mem addr: 0x04b6, content: 0xff Flash mem addr: 0x04b7, content: 0x36 Flash mem addr: 0x04b8, content: 0xff Flash mem addr: 0x04b9, content: 0x38 Flash mem addr: 0x04ba, content: 0xff Flash mem addr: 0x04bb, content: 0x3a Flash mem addr: 0x04bc, content: 0xff Flash mem addr: 0x04bd, content: 0x3c Flash mem addr: 0x04be, content: 0xff Flash mem addr: 0x04bf, content: 0x3e Flash mem addr: 0x04c0, content: 0xff Flash mem addr: 0x04c1, content: 0x40 Flash mem addr: 0x04c2, content: 0xff Flash mem addr: 0x04c3, content: 0x42 Flash mem addr: 0x04c4, content: 0xff Flash mem addr: 0x04c5, content: 0x44 Flash mem addr: 0x04c6, content: 0xff Flash mem addr: 0x04c7, content: 0x46 Flash mem addr: 0x04c8, content: 0xff Flash mem addr: 0x04c9, content: 0x48 Flash mem addr: 0x04ca, content: 0xff Flash mem addr: 0x04cb, content: 0x4a Flash mem addr: 0x04cc, content: 0xff Flash mem addr: 0x04cd, content: 0x4c Flash mem addr: 0x04ce, content: 0xff Flash mem addr: 0x04cf, content: 0x4e Flash mem addr: 0x04d0, content: 0xff Flash mem addr: 0x04d1, content: 0x50 Flash mem addr: 0x04d2, content: 0xff Flash mem addr: 0x04d3, content: 0x52 Flash mem addr: 0x04d4, content: 0xff Flash mem addr: 0x04d5, content: 0x54 Flash mem addr: 0x04d6, content: 0xff Flash mem addr: 0x04d7, content: 0x56 Flash mem addr: 0x04d8, content: 0xff Flash mem addr: 0x04d9, content: 0x58 Flash mem addr: 0x04da, content: 0xff Flash mem addr: 0x04db, content: 0x5a Flash mem addr: 0x04dc, content: 0xff Flash mem addr: 0x04dd, content: 0x5c Flash mem addr: 0x04de, content: 0xff Flash mem addr: 0x04df, content: 0x5e Flash mem addr: 0x04e0, content: 0xff Flash mem addr: 0x04e1, content: 0x60 Flash mem addr: 0x04e2, content: 0xff Flash mem addr: 0x04e3, content: 0x62 Flash mem addr: 0x04e4, content: 0xff Flash mem addr: 0x04e5, content: 0x64 Flash mem addr: 0x04e6, content: 0xff Flash mem addr: 0x04e7, content: 0x66 Flash mem addr: 0x04e8, content: 0xff Flash mem addr: 0x04e9, content: 0x68 Flash mem addr: 0x04ea, content: 0xff Flash mem addr: 0x04eb, content: 0x6a Flash mem addr: 0x04ec, content: 0xff Flash mem addr: 0x04ed, content: 0x6c Flash mem addr: 0x04ee, content: 0xff Flash mem addr: 0x04ef, content: 0x6e Flash mem addr: 0x04f0, content: 0xff Flash mem addr: 0x04f1, content: 0x70 Flash mem addr: 0x04f2, content: 0xff Flash mem addr: 0x04f3, content: 0x72 Flash mem addr: 0x04f4, content: 0xff Flash mem addr: 0x04f5, content: 0x74 Flash mem addr: 0x04f6, content: 0xff Flash mem addr: 0x04f7, content: 0x76 Flash mem addr: 0x04f8, content: 0xff Flash mem addr: 0x04f9, content: 0x78 Flash mem addr: 0x04fa, content: 0xff Flash mem addr: 0x04fb, content: 0x7a Flash mem addr: 0x04fc, content: 0xff Flash mem addr: 0x04fd, content: 0x7c Flash mem addr: 0x04fe, content: 0xff Flash mem addr: 0x04ff, content: 0x7e ```

Another thing. I know these chips have "memory-mapped flash", but I don't know exactly how this works on a very low level. I've not been able to get rid of the PROGMEM attribute or pgm_read_byte, and I'm not sure if this is possible or not. If yes, it will reduce the compiled size of the demo sketches.

technoblogy commented 3 years ago

Do you have any experience with the "modern" AVRs and "write-to-flash" functionality?

I've used Spence Konde's DxCore Flash Writer on the AVR128DA/DB and that works well, without needing a special bootloader. Wouldn't the same approach work on the ATmega4809 etc?

MCUdude commented 3 years ago

BTW I solved my initial issue! Working example code can be found here.

I've used Spence Konde's DxCore Flash Writer on the AVR128DA/DB and that works well, without needing a special bootloader. Wouldn't the same approach work on the ATmega4809 etc?

That's right, I didn't think of this. Haven't looked into this for the megaAVR-0 series yet, but I did take a look at @SpenceKonde's source code. What's nice about having this functionality in the bootloader is that it "just works". I tried to copy the ´do_nvmctrl()´ function found in the bootloader and into my own code, and that didn't work at all. So there still seems to be some kind of bootloader space magic happening. Yet, Spence has found a way around it.

But with DxCore it seems like you'll have to select in the Arduino Tools where in flash you want to have flash write functionality. Why wouldn't you have this anywhere, like with the bootloader implementation? What are the downsides?
For a non-assembly guy like me, the code overall seems very cryptic to me too. Does there really have to be inline assembly in there? How can the Optiboot do_nvmctrl implementation be so simple, while the NVMCTRL code for the DA/DB series be so complex in comparison?

SpenceKonde commented 3 years ago

There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails). When we are not using the bootloader, and hence can adjust the CODESIZE fuse with every upload we can use that to make only the section of the flash that the user wishes to expose to writes writable (and we avoid needing to have a piece of code running from the bootloader section). Note that this secctiuoion should be sized such that all of your code is within codesize ( I belive this is in the docs), because code that passes outside of CODESIZE in the limited-writable-area configuration would be running from APPDATA and can't write to flash or EEPROM.

When "anywhere" is selected, CODESIZE is 0, bootsize is the minimum, and the 4 byte magic thing-to-jump-to is placed in the trampolines section of the flash. located immediately after the vector table, hence ensuring it's within first (bootloader) page of flash. This gives it it full access to the rest of flash. But it can now scribble over any part of the flash.

Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs. Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine. As it happens, I was able to achieve full code sharing between the three cases in Flash.cpp - the only things that change in flash.cpp between the three options (optiboot, anywhere no optiboot, no optiboot restricted location) is that with anywhere restricted, it's a straight up 'spm z+' in the assembly, whereas for the others, it's a call instruction, either to the name of the label for the spm entrypoint (for anywhere no optiboot), or to 0x2FA (512 - 6 bytes, ie, 3 words before the end of the bootloader section on optiboot configurations. those last three words are spm z+, ret,. and the optiboot serial number)

Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..

How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?

I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes uise of it to write to flash from the application is more complicated by a significant margin.

Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(

technoblogy commented 3 years ago

@SpenceKonde thanks for the info. To check I've understood correctly:

MCUdude commented 3 years ago

There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails).

I don't see why this is so "dangerous" really. As long as you keep yourself inside a pre-defined array like this there are no risks. The library even forces you to it. Heck, you can screw up your programs with pointers too, and you can't protect yourself from this with fuses, just good coding practice.

Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs.

Did you write the inline assembly yourself, or did you find it in some Microchip application note. I'd like to know more about the topic in general, so if you know some good documentation apart from the datasheet, please let me know!

Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine.

I believe Optiboot for the Dx series and Optiboot for the megaAVR-0 differs then. The implementation for megaAVR-0 does not look like a hack job at all IMO.

Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..

So this means that Writing to flash from within application, without using Optiboot is pretty much impossible for the megaAVR-0 and tiny0/1/2 series?

How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?

I do now realize it has to look like the way it does, with the inline assembly and everything. Originally, I was comparing the "dead simple" do_nvmctrl function found inside the megaAVR-0 bootloader and the write routine you've provided in Flash.cpp.

Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(

Well, not really. It's just that I lack the resources to figure out exactly what's going on. But I don't have a clue what's going on in the "classic" boot.h either 😅

I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes use of it to write to flash from the application is more complicated by a significant margin.

I'm not sure what example you're referring to. Link please?

When it comes to wrapping the low-level commands in a library, and make them accessible to the average semi-advanced Arduino user, I feel like this is something I'm reasonably good at. You should take a look at the Flash library I recently push, and the Flash_put_get example where you can just throw anything into the flash memory and you can retrieve it the same way. I've borrowed some template-based code from the Arduino EEPROM library, and that's what makes put() and get() so versatile.

SpenceKonde commented 3 years ago

@SpenceKonde thanks for the info. To check I've understood correctly:

  • On AVR DA/DB, user code can write to the flash either without or with a special bootloader (in DxCore).

It can write to the flash as long as BOOTSIZE is not 0 You don't need to have there actually BE a bootloader - you can just have init() set the bit in the interrupt controller register to move the vectors back to the "bootloader section" (which is where they will be if you compile normally, which you presumable will). If BOOTSIZE is anything other than 0, there are three flash sections.

This is actually the same on tinyAVR 0/1/2 and megaAVR 0 - though they named the fuses differently, they function identically.

0 ~ BOOTSIZE*512 is BOOTCODE section. It can never be selfprogrammed, ever. 
Only writable over UPDI. 
Code running from here can write to APPCODE and APPDATA.

BOOTSIZE*512 ~ CODESIZE*512 is APPCODE section. 
It can be written to by code running from the BOOTCODE section, 
It can write to EEPROM, and to flash located in the APPDATA section. 

CODESIZE*512 to end of flash is APPDATA section. 
Code executing from here cannot write to EEPROM or flash. 

If BOOTSIZE = 0 (no bootloader) then ALL FLASH is treated as BOOTCODE. 
No self-programming is possible in this case. 

There are four cases that DxCore supports:

  • On old ATmega, eg ATmega1284P, user code can only write to the flash with a special bootloader (in MightyCore).

Correct, if a bootloader was specified by the fuses, self programming could only be carried out from the bootloader flash.

  • On new ATtiny 0/1/2-series (megaTinyCore) and new ATmega 0-series (MegaCoreX), user code can only write to the flash with a special bootloader, but you haven't figured out how to do it yet?

It could be done in a theoretically similar way to what I did for DxCore, but both writes to the temporary buffer, AND writes to the NVMCTRL registers are protected; it would be much less convenient. Since code has been written to write to flash from app through the bootloader, I have better things to do than implementing another approach to it. The only catch is that I don't know how to use the optiboot implementation.

Note re: bootloader size: because when you compile code, it has to be aware of the size of the bootloader so it knows what address to start from, and because that start is the vector table; bootloaders of different sizes are now completely, totally, 100% incompatible now - whereas on classuic parts, they were totally compatible unlless you expected to use flash containing the bootloader.

SpenceKonde commented 3 years ago

There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails).

I don't see why this is so "dangerous" really. As long as you keep yourself inside a pre-defined array like this there are no risks. The library even forces you to it. Heck, you can screw up your programs with pointers too, and you can't protect yourself from this with fuses, just good coding practice.

Not arguing with you, but like if you're doing it based on user input, or if the input is coming from the internet (ie, check the config file), it could give extra peace of mind vs having tro be REALLY sure that your parser rejects bad incoming data....

Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs.

Did you write the inline assembly yourself, or did you find it in some Microchip application note. I'd like to know more about the topic in general, so if you know some good documentation apart from the datasheet, please let me know!

I write it myself. My main resource is the bible https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-Instruction-Set-Manual-DS40002198A.pdf - like the real world bible, it contains truth revealed by the creator, cloaked in obtuse verbiage (be sure you use that version, not the one with Atmel branding - there are some rather subtle points on which it can lead you astray, and the microchip branded one is... just done better. ), plus the inline assembly cookbook in the avrgcc docs, and this avrdude thread is also a useful reference. https://www.avrfreaks.net/forum/few-remarks-avr-gcc-inline-assembler Assembly is really very simple, IMO - but because it does nothing for you, it is very very slow to write. I find it harder to write bugs in than C - but much harder to fix bugs in.

Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine.

I'm referring to how optiboot.h gets execution to the bootloader in all of those methods. Haven't looked at it in a few months, but I don't think it's any more graceful than inline assembly specifying the address to jump to.

I believe Optiboot for the Dx series and Optiboot for the megaAVR-0 differs then. The implementation for megaAVR-0 does not look like a hack job at all IMO.

Not talking about the part inside the bootloader (though now that you bring it up, https://github.com/MCUdude/MegaCoreX/blob/5331a498a4e78ece00eafe8961bffbdae900f3a3/megaavr/bootloaders/optiboot/optiboot_x.c#L288-L301 ) - the implementation I put into optiboot dx for the entry point is way dirtier . - it's just an unsigned long with attribute ((used)) and the correct numeric vbalue placed immediately before the version. But it doesn't waste any flash, and like I said in previous message I was desperate for a couple of bytes at the end,...

Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..

So this means that Writing to flash from within application, without using Optiboot is pretty much impossible for the megaAVR-0 and tiny0/1/2 series?

A scheme like what I did is possible, but needs more complicated code stuffed into the bootloader-blessed section of flash

How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?

I do now realize it has to look like the way it does, with the inline assembly and everything. Originally, I was comparing the "dead simple" do_nvmctrl function found inside the megaAVR-0 bootloader and the write routine you've provided in Flash.cpp.

Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(

Well, not really. It's just that I lack the resources to figure out exactly what's going on. But I don't have a clue what's going on in the "classic" boot.h either 😅

I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes use of it to write to flash from the application is more complicated by a significant margin.

I'm not sure what example you're referring to. Link please?

When it comes to wrapping the low-level commands in a library, and make them accessible to the average semi-advanced Arduino user, I feel like this is something I'm reasonably good at. You should take a look at the Flash library I recently push, and the Flash_put_get example where you can just throw anything into the flash memory and you can retrieve it the same way. I've borrowed some template-based code from the Arduino EEPROM library, and that's what makes put() and get() so versatile.

it's the example in the optiboot repo, don't have time to find it, need to do what I was doing when I started typing these rteplies out.

technoblogy commented 3 years ago

There's a slightly more recent AVR Instruction Set Manual:

https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf

Don't know whether it's different.

SpenceKonde commented 3 years ago

There's a slightly more recent AVR Instruction Set Manual:

https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf

Don't know whether it's different.

So there is! Glory to Microchip!

Also, goddamnit, they slipped up in the device table again (last time, they listed Dx series as AVRxm, not AVRxt), they have the DD listed with 128, 64, and 32k of flash. It has been announced for 64, 32, and 16k, and since it's meant to be a cut-down DB, I would be very surprised if that table was right and the rest of what they've said was wrong.

MCUdude commented 3 years ago

@SpenceKonde in the last commit I've removed the need for the PROGMEM attribute, which makes the code even more elegant for the megaAVR-0 and the tiny-0/1/2's. However, PROGMEM still needs to be present if this library would be ported to the Dx series

SpenceKonde commented 3 years ago

Thanks for letting me know - I don't plan to port to Dx. For one thing, the Dx optiboot doesn';t have do_spm. There isn't enough flash for it - I barely had the 4 bytes my entry point takes.... I was implementing larger and larger chunks of tjhe bootloader in inline assembly to save 1-2 words of flash per chunk so I could get it to make it respect entry conditions

MCUdude commented 3 years ago

@SpenceKonde It wouldn't be much work porting it to DxCore. All you need to do is to replace optiboot_read() and optiboot_WritePage() with your own functions in Flash.cpp.

SpenceKonde commented 3 years ago

That may be the case, but not porting a library when we already have a library for writing to flash from app is zero work, and I am desperately behind in everything. The optiboot flash writier is being pulled in only because I think it should just work, otherwise it would be a low priority and get deferred indefinitely until someone or sometthing applied pressure on me to get it done.

On Sun, Apr 11, 2021 at 6:05 AM Hans @.***> wrote:

@SpenceKonde https://github.com/SpenceKonde It wouldn't be much work porting it to DxCore. All you need to do is to replace optiboot_read() and optiboot_WritePage() with your own functions in Flash.cpp https://github.com/MCUdude/MegaCoreX/blob/5be3f30fbd196bc9dee0ecc660a6f7d64b95024b/megaavr/libraries/Optiboot_flasher/src/Flash.cpp#L109-L182 .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MCUdude/MegaCoreX/issues/116#issuecomment-817281891, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTXEW4PE3ZLQ4LTSXKSJ2LTIFX63ANCNFSM42MSWD3Q .

--


Spence Konde Azzy’S Electronics

New products! Check them out at tindie.com/stores/DrAzzy GitHub: github.com/SpenceKonde ATTinyCore https://github.com/SpenceKonde/ATTinyCore: Arduino support for all pre-2016 tinyAVR with >2k flash! megaTinyCore https://github.com/SpenceKonde/megaTinyCore: Arduino support for all post-2016 tinyAVR parts! DxCore https://github.com/SpenceKonde/DxCore: Arduino support for the AVR Dx-series parts, the latest and greatest from Microchip! Contact: @.***

SpenceKonde commented 3 years ago

Was just looking at bringing this into megaTinyCore. Where is the documentation? There's not even a readme... :-/ Which if you'll recall, is why optiboot.h itself was never packaged with my core!

MCUdude commented 3 years ago

I've not created a readme yet, only provided examples and commenting the code in the cpp files. The comments are "Doxygen styled" so the comment gets nicely formatted if you're using a modern editor.

The code itself is not particularly hard to figure out though. It's just read from flash and do_nvmctrl really. So you're not interested because I haven't prepared everything this time?

SpenceKonde commented 3 years ago

No, it'll go in - but there will be no documentation with it. It definitely is more comprehensible than straight optiboot.h was, and the examples are better. I don't like putting in libraries without documentation, but is certainly not something to not include a library over.

MCUdude commented 3 years ago

Give me a few days and I'll see if I can create some documentation for the libraries

MCUdude commented 3 years ago

@SpenceKonde Here's a README:

https://github.com/MCUdude/MegaCoreX/blob/master/megaavr/libraries/Optiboot_flasher/README.md

SpenceKonde commented 3 years ago

Hmm, I just noticed these lines while changing formatting to match the style of the rest of the core (it's enforced by the stylechecker too_)

| IMPORTANT THINGS: | | - All flash content gets erased after each upload cycle |

I don't think that's true if the upload is done using optiboot - which it would have to be if this library is in use, Optoiboot erases pages as it fills them. upload a small sketch after having a large sketch in there, and then read back the flash, and you'll find a bunch of debris after your code.

MCUdude commented 3 years ago

I don't think that's true if the upload is done using optiboot - which it would have to be if this library is in use, Optoiboot erases pages as it fills them. upload a small sketch after having a large sketch in there, and then read back the flash, and you'll find a bunch of debris after your code.

Interesting. Have you tested and verified this? And what would happen if the allocated space was declared PROGMEM and initialized as 0x00?

SpenceKonde commented 3 years ago

Not sure if I have on tinyAVR 0/1-serties (and certainly not on megaAVR 0-series) but you can see in optiboot_x.c there's no facility to erase the whole memory, it just does the pages that it's writing the new upload to using NVMCTRL_CMD_PAGEERASEWRITE_gc.

Space allocated via a declareation like that would of course be set to 0 on every upload, as it's part of the hex file (I think progmem also forces the code to be put near the beginning of the flash, before most code instead of after; not sure if that happens on all parts though). I've always recommended using flash from the end of the flash when writing from the appliction if one has the freedom to choose where to put it.

MCUdude commented 3 years ago

Not sure if I have on tinyAVR 0/1-serties (and certainly not on megaAVR 0-series) but you can see in optiboot_x.c there's no facility to erase the whole memory, it just does the pages that it's writing the new upload to using NVMCTRL_CMD_PAGEERASEWRITE_gc.

So you mean it won't write zeros to the flash memory even if it's in the hex file? I just compared a sketch where I've allocated 2 and 20 pages. 20 pages leave a lot of zeros on the flash. I would assume these zeros would be written to flash.

SpenceKonde commented 3 years ago

No no, if it's allocated with a progmem declaration, that would always be written.

But if it's not declared (as in, you just pick an address and start reading and writing it - which is how I approach flash writes from within app) it will persist between uploads unless you upload something that overlaps with it.

MCUdude commented 3 years ago

Yes, if the user just starts to read/write to arbitrary addresses, chances are that the flash space has some remains from old programs. But the whole idea with the library is that flash space is pre-allocated, and that the user has to stay within this space