Closed MCUdude closed 3 years ago
Do you have any experience with the "modern" AVRs and "write-to-flash" functionality?
I've used Spence Konde's DxCore Flash Writer on the AVR128DA/DB and that works well, without needing a special bootloader. Wouldn't the same approach work on the ATmega4809 etc?
BTW I solved my initial issue! Working example code can be found here.
I've used Spence Konde's DxCore Flash Writer on the AVR128DA/DB and that works well, without needing a special bootloader. Wouldn't the same approach work on the ATmega4809 etc?
That's right, I didn't think of this. Haven't looked into this for the megaAVR-0 series yet, but I did take a look at @SpenceKonde's source code. What's nice about having this functionality in the bootloader is that it "just works". I tried to copy the ´do_nvmctrl()´ function found in the bootloader and into my own code, and that didn't work at all. So there still seems to be some kind of bootloader space magic happening. Yet, Spence has found a way around it.
But with DxCore it seems like you'll have to select in the Arduino Tools where in flash you want to have flash write functionality. Why wouldn't you have this anywhere, like with the bootloader implementation? What are the downsides?
For a non-assembly guy like me, the code overall seems very cryptic to me too. Does there really have to be inline assembly in there? How can the Optiboot do_nvmctrl implementation be so simple, while the NVMCTRL code for the DA/DB series be so complex in comparison?
There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails). When we are not using the bootloader, and hence can adjust the CODESIZE fuse with every upload we can use that to make only the section of the flash that the user wishes to expose to writes writable (and we avoid needing to have a piece of code running from the bootloader section). Note that this secctiuoion should be sized such that all of your code is within codesize ( I belive this is in the docs), because code that passes outside of CODESIZE in the limited-writable-area configuration would be running from APPDATA and can't write to flash or EEPROM.
When "anywhere" is selected, CODESIZE is 0, bootsize is the minimum, and the 4 byte magic thing-to-jump-to is placed in the trampolines section of the flash. located immediately after the vector table, hence ensuring it's within first (bootloader) page of flash. This gives it it full access to the rest of flash. But it can now scribble over any part of the flash.
Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs. Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine. As it happens, I was able to achieve full code sharing between the three cases in Flash.cpp - the only things that change in flash.cpp between the three options (optiboot, anywhere no optiboot, no optiboot restricted location) is that with anywhere restricted, it's a straight up 'spm z+' in the assembly, whereas for the others, it's a call instruction, either to the name of the label for the spm entrypoint (for anywhere no optiboot), or to 0x2FA (512 - 6 bytes, ie, 3 words before the end of the bootloader section on optiboot configurations. those last three words are spm z+, ret,. and the optiboot serial number)
Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..
How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?
I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes uise of it to write to flash from the application is more complicated by a significant margin.
Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(
@SpenceKonde thanks for the info. To check I've understood correctly:
There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails).
I don't see why this is so "dangerous" really. As long as you keep yourself inside a pre-defined array like this there are no risks. The library even forces you to it. Heck, you can screw up your programs with pointers too, and you can't protect yourself from this with fuses, just good coding practice.
Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs.
Did you write the inline assembly yourself, or did you find it in some Microchip application note. I'd like to know more about the topic in general, so if you know some good documentation apart from the datasheet, please let me know!
Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine.
I believe Optiboot for the Dx series and Optiboot for the megaAVR-0 differs then. The implementation for megaAVR-0 does not look like a hack job at all IMO.
Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..
So this means that Writing to flash from within application, without using Optiboot is pretty much impossible for the megaAVR-0 and tiny0/1/2 series?
How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?
I do now realize it has to look like the way it does, with the inline assembly and everything. Originally, I was comparing the "dead simple" do_nvmctrl
function found inside the megaAVR-0 bootloader and the write routine you've provided in Flash.cpp.
Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(
Well, not really. It's just that I lack the resources to figure out exactly what's going on. But I don't have a clue what's going on in the "classic" boot.h either 😅
I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes use of it to write to flash from the application is more complicated by a significant margin.
I'm not sure what example you're referring to. Link please?
When it comes to wrapping the low-level commands in a library, and make them accessible to the average semi-advanced Arduino user, I feel like this is something I'm reasonably good at. You should take a look at the Flash library I recently push, and the Flash_put_get example where you can just throw anything into the flash memory and you can retrieve it the same way. I've borrowed some template-based code from the Arduino EEPROM library, and that's what makes put()
and get()
so versatile.
@SpenceKonde thanks for the info. To check I've understood correctly:
- On AVR DA/DB, user code can write to the flash either without or with a special bootloader (in DxCore).
It can write to the flash as long as BOOTSIZE is not 0 You don't need to have there actually BE a bootloader - you can just have init() set the bit in the interrupt controller register to move the vectors back to the "bootloader section" (which is where they will be if you compile normally, which you presumable will). If BOOTSIZE is anything other than 0, there are three flash sections.
This is actually the same on tinyAVR 0/1/2 and megaAVR 0 - though they named the fuses differently, they function identically.
0 ~ BOOTSIZE*512 is BOOTCODE section. It can never be selfprogrammed, ever.
Only writable over UPDI.
Code running from here can write to APPCODE and APPDATA.
BOOTSIZE*512 ~ CODESIZE*512 is APPCODE section.
It can be written to by code running from the BOOTCODE section,
It can write to EEPROM, and to flash located in the APPDATA section.
CODESIZE*512 to end of flash is APPDATA section.
Code executing from here cannot write to EEPROM or flash.
If BOOTSIZE = 0 (no bootloader) then ALL FLASH is treated as BOOTCODE.
No self-programming is possible in this case.
There are four cases that DxCore supports:
No Optiboot, No flash write. BOOTSIZE = 0, no self programming.
Yes Optiboot. BOOTSIZE = 1, Optioboot crammed into 512b (beyond my distaste for do_nvmctrl(), we didn't have the flash to do that on 128k parts without either going to a 1k bootloader (which we could do we have plenty of flash but one really doens't want to do that if you might later put out a version that does fit in 512b (see note below) - I was getting fucking desperate man, I was rewriting large and larger chunks of the flash write part in assembly and then being like FUUUUUCK all that work to save only 2 words? (happened every time - was never able to save as much as I thought I could - the compiler insists on doing dumb stuff, even when you try to manually do it intelligently, it finds ways to undo your work). When I realized I could axe do_nvmctrl's inefficient hook and replace it with 2 instruction words at the very end, not only did it fit, but I was able to fix almost every entry-condition hazard (corner - or not-so-corner - cases where the "In this case enter optiboot, otherwise start app" can get it wrong if not careful). When writing from app, to execute SPM instruction, we "call 0x1fa" ( 3 instructions before end of bootloader) which is where i stuffed the magic write-from-app instructions (spm z+ ret) - only takes 2 instructions/4 bytes!). Nothing else requires running from boot
No Optiboot, Limited flash write area. BOOTSIZE = 1, CODESIZE per writable area option (and at the very beginning, (I think I do it even before init), I do CPUINT.CTRLA = IVSEL_bm to tell it that the vectors start at 0, not 512. the first 512b are "bootloader" flash, but it has no impact on operation. After CODESIZE*512, it is all APPDATA and the sketch can write to it freely. Sketch cannot overwrite itself (in the APPCODE section). When writing from app, we execute SPM directly, nothing wacky.
No optiboot, flash write permitted anywhere. BOOTSIZE = 1 CODESZE = 0 - first 512b are BOOTCODE rest is APPCODE Now, APPCODE can't write to other APPCODE, so we need to find a way to get an spm z+ret
into the first 512b. This took me a while to figure out; my solution was to locate it in the trampolines section, as a 3-word sequence: rjmp .+4 (if execution reaches here, we hop over the next 2 words (4 bytes - the assembler expects bytes for everything, even though it's all expressed as words in the opcode, not bytes), followed by the two key instructions. I don't know what;'s going on around there, but without that guard, the code never ran, so apparently execution did pass through there! I had originally planned to put it in an unused vector (with 2 word vectors like these parts, if you have one you know will never execute (I was gonna use the NMI) you could replace the 2-word jmp with spm z+ and ret, but that turned out to not be practical, because I would have needed to build a new CRT and make the compiler use it! so stuffed into an early section (the purpose of which I do not understand), with a "jump over me"
instruction at the start it had to be.
- On old ATmega, eg ATmega1284P, user code can only write to the flash with a special bootloader (in MightyCore).
Correct, if a bootloader was specified by the fuses, self programming could only be carried out from the bootloader flash.
- On new ATtiny 0/1/2-series (megaTinyCore) and new ATmega 0-series (MegaCoreX), user code can only write to the flash with a special bootloader, but you haven't figured out how to do it yet?
It could be done in a theoretically similar way to what I did for DxCore, but both writes to the temporary buffer, AND writes to the NVMCTRL registers are protected; it would be much less convenient. Since code has been written to write to flash from app through the bootloader, I have better things to do than implementing another approach to it. The only catch is that I don't know how to use the optiboot implementation.
Note re: bootloader size: because when you compile code, it has to be aware of the size of the bootloader so it knows what address to start from, and because that start is the vector table; bootloaders of different sizes are now completely, totally, 100% incompatible now - whereas on classuic parts, they were totally compatible unlless you expected to use flash containing the bootloader.
There is an "anywhere" option in the dropdown menu - you'e thinking about that part backwards; When we have an optiboot-boot;loaded board, because we can't set fuses through the bootloader, we do not have the option to restrict writing to a specific section of the flash with the CODESIZE fuse. Thus, all we can offer the user is an option to write anywhere except the bootloader (no guard rails).
I don't see why this is so "dangerous" really. As long as you keep yourself inside a pre-defined array like this there are no risks. The library even forces you to it. Heck, you can screw up your programs with pointers too, and you can't protect yourself from this with fuses, just good coding practice.
Not arguing with you, but like if you're doing it based on user input, or if the input is coming from the internet (ie, check the config file), it could give extra peace of mind vs having tro be REALLY sure that your parser rejects bad incoming data....
Yes,. there absolutely does need to be inline assembly in there. The SPM instruction CANNOT BE INVOKED without inline assembly, and the only reason there are any bootloaders that don't have inline assembly is that there used to be that boot.h file from avrlibc you could use which had the inline assembly there, but it hasn't been updated for the new AVRs.
Did you write the inline assembly yourself, or did you find it in some Microchip application note. I'd like to know more about the topic in general, so if you know some good documentation apart from the datasheet, please let me know!
I write it myself. My main resource is the bible https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-Instruction-Set-Manual-DS40002198A.pdf - like the real world bible, it contains truth revealed by the creator, cloaked in obtuse verbiage (be sure you use that version, not the one with Atmel branding - there are some rather subtle points on which it can lead you astray, and the microchip branded one is... just done better. ), plus the inline assembly cookbook in the avrgcc docs, and this avrdude thread is also a useful reference. https://www.avrfreaks.net/forum/few-remarks-avr-gcc-inline-assembler Assembly is really very simple, IMO - but because it does nothing for you, it is very very slow to write. I find it harder to write bugs in than C - but much harder to fix bugs in.
Optiboot.h uses some unholy hackjob to make the compiler generate a call into the bootloader section - I think that approach is no better than mine.
I'm referring to how optiboot.h gets execution to the bootloader in all of those methods. Haven't looked at it in a few months, but I don't think it's any more graceful than inline assembly specifying the address to jump to.
I believe Optiboot for the Dx series and Optiboot for the megaAVR-0 differs then. The implementation for megaAVR-0 does not look like a hack job at all IMO.
Not talking about the part inside the bootloader (though now that you bring it up, https://github.com/MCUdude/MegaCoreX/blob/5331a498a4e78ece00eafe8961bffbdae900f3a3/megaavr/bootloaders/optiboot/optiboot_x.c#L288-L301 ) - the implementation I put into optiboot dx for the entry point is way dirtier . - it's just an unsigned long with attribute ((used)) and the correct numeric vbalue placed immediately before the version. But it doesn't waste any flash, and like I said in previous message I was desperate for a couple of bytes at the end,...
Apparently on the tinyAVR 0/1/2 and mega0 you had to be running from bootloader to both fill the page buffer AND operate the nvmctrl register to command it. Here on the other hand, it's dead simple: Only the SPM instruction, or an ST instruction which writes to NVM, are protected. Everything else can be done by code running from any location in the flash..
So this means that Writing to flash from within application, without using Optiboot is pretty much impossible for the megaAVR-0 and tiny0/1/2 series?
A scheme like what I did is possible, but needs more complicated code stuffed into the bootloader-blessed section of flash
How do you say the nvmctrl part of the DA/DB code is complicated? What do you find complicated about it?
I do now realize it has to look like the way it does, with the inline assembly and everything. Originally, I was comparing the "dead simple"
do_nvmctrl
function found inside the megaAVR-0 bootloader and the write routine you've provided in Flash.cpp.Were the comments in Flash.cpp that bad? I tried to do a decent job of commenting it :-(
Well, not really. It's just that I lack the resources to figure out exactly what's going on. But I don't have a clue what's going on in the "classic" boot.h either 😅
I;'ve stared at the optiboot flash write code for tinyAVR 0/1/2 and tried to figure out how to write a library to expose it within Arduino. Have not been successful. I can't follow what happens at a higher level than the call into the bootloader.... . It is totally undocumented, and they just refer to a complicated example that has so much functionality that it has obscured the API. I think both the nvmctrl implementation in hardware, and the way optiboot makes use of it to write to flash from the application is more complicated by a significant margin.
I'm not sure what example you're referring to. Link please?
When it comes to wrapping the low-level commands in a library, and make them accessible to the average semi-advanced Arduino user, I feel like this is something I'm reasonably good at. You should take a look at the Flash library I recently push, and the Flash_put_get example where you can just throw anything into the flash memory and you can retrieve it the same way. I've borrowed some template-based code from the Arduino EEPROM library, and that's what makes
put()
andget()
so versatile.
it's the example in the optiboot repo, don't have time to find it, need to do what I was doing when I started typing these rteplies out.
There's a slightly more recent AVR Instruction Set Manual:
https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf
Don't know whether it's different.
There's a slightly more recent AVR Instruction Set Manual:
https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf
Don't know whether it's different.
So there is! Glory to Microchip!
Also, goddamnit, they slipped up in the device table again (last time, they listed Dx series as AVRxm, not AVRxt), they have the DD listed with 128, 64, and 32k of flash. It has been announced for 64, 32, and 16k, and since it's meant to be a cut-down DB, I would be very surprised if that table was right and the rest of what they've said was wrong.
@SpenceKonde in the last commit I've removed the need for the PROGMEM attribute, which makes the code even more elegant for the megaAVR-0 and the tiny-0/1/2's. However, PROGMEM still needs to be present if this library would be ported to the Dx series
Thanks for letting me know - I don't plan to port to Dx. For one thing, the Dx optiboot doesn';t have do_spm. There isn't enough flash for it - I barely had the 4 bytes my entry point takes.... I was implementing larger and larger chunks of tjhe bootloader in inline assembly to save 1-2 words of flash per chunk so I could get it to make it respect entry conditions
@SpenceKonde It wouldn't be much work porting it to DxCore. All you need to do is to replace optiboot_read()
and optiboot_WritePage()
with your own functions in Flash.cpp.
That may be the case, but not porting a library when we already have a library for writing to flash from app is zero work, and I am desperately behind in everything. The optiboot flash writier is being pulled in only because I think it should just work, otherwise it would be a low priority and get deferred indefinitely until someone or sometthing applied pressure on me to get it done.
On Sun, Apr 11, 2021 at 6:05 AM Hans @.***> wrote:
@SpenceKonde https://github.com/SpenceKonde It wouldn't be much work porting it to DxCore. All you need to do is to replace optiboot_read() and optiboot_WritePage() with your own functions in Flash.cpp https://github.com/MCUdude/MegaCoreX/blob/5be3f30fbd196bc9dee0ecc660a6f7d64b95024b/megaavr/libraries/Optiboot_flasher/src/Flash.cpp#L109-L182 .
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MCUdude/MegaCoreX/issues/116#issuecomment-817281891, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTXEW4PE3ZLQ4LTSXKSJ2LTIFX63ANCNFSM42MSWD3Q .
--
Spence Konde Azzy’S Electronics
New products! Check them out at tindie.com/stores/DrAzzy GitHub: github.com/SpenceKonde ATTinyCore https://github.com/SpenceKonde/ATTinyCore: Arduino support for all pre-2016 tinyAVR with >2k flash! megaTinyCore https://github.com/SpenceKonde/megaTinyCore: Arduino support for all post-2016 tinyAVR parts! DxCore https://github.com/SpenceKonde/DxCore: Arduino support for the AVR Dx-series parts, the latest and greatest from Microchip! Contact: @.***
Was just looking at bringing this into megaTinyCore. Where is the documentation? There's not even a readme... :-/ Which if you'll recall, is why optiboot.h itself was never packaged with my core!
I've not created a readme yet, only provided examples and commenting the code in the cpp files. The comments are "Doxygen styled" so the comment gets nicely formatted if you're using a modern editor.
The code itself is not particularly hard to figure out though. It's just read from flash and do_nvmctrl really. So you're not interested because I haven't prepared everything this time?
No, it'll go in - but there will be no documentation with it. It definitely is more comprehensible than straight optiboot.h was, and the examples are better. I don't like putting in libraries without documentation, but is certainly not something to not include a library over.
Give me a few days and I'll see if I can create some documentation for the libraries
@SpenceKonde Here's a README:
https://github.com/MCUdude/MegaCoreX/blob/master/megaavr/libraries/Optiboot_flasher/README.md
Hmm, I just noticed these lines while changing formatting to match the style of the rest of the core (it's enforced by the stylechecker too_)
| IMPORTANT THINGS: | | - All flash content gets erased after each upload cycle |
I don't think that's true if the upload is done using optiboot - which it would have to be if this library is in use, Optoiboot erases pages as it fills them. upload a small sketch after having a large sketch in there, and then read back the flash, and you'll find a bunch of debris after your code.
I don't think that's true if the upload is done using optiboot - which it would have to be if this library is in use, Optoiboot erases pages as it fills them. upload a small sketch after having a large sketch in there, and then read back the flash, and you'll find a bunch of debris after your code.
Interesting. Have you tested and verified this? And what would happen if the allocated space was declared PROGMEM and initialized as 0x00?
Not sure if I have on tinyAVR 0/1-serties (and certainly not on megaAVR 0-series) but you can see in optiboot_x.c there's no facility to erase the whole memory, it just does the pages that it's writing the new upload to using NVMCTRL_CMD_PAGEERASEWRITE_gc.
Space allocated via a declareation like that would of course be set to 0 on every upload, as it's part of the hex file (I think progmem also forces the code to be put near the beginning of the flash, before most code instead of after; not sure if that happens on all parts though). I've always recommended using flash from the end of the flash when writing from the appliction if one has the freedom to choose where to put it.
Not sure if I have on tinyAVR 0/1-serties (and certainly not on megaAVR 0-series) but you can see in optiboot_x.c there's no facility to erase the whole memory, it just does the pages that it's writing the new upload to using NVMCTRL_CMD_PAGEERASEWRITE_gc.
So you mean it won't write zeros to the flash memory even if it's in the hex file? I just compared a sketch where I've allocated 2 and 20 pages. 20 pages leave a lot of zeros on the flash. I would assume these zeros would be written to flash.
No no, if it's allocated with a progmem declaration, that would always be written.
But if it's not declared (as in, you just pick an address and start reading and writing it - which is how I approach flash writes from within app) it will persist between uploads unless you upload something that overlaps with it.
Yes, if the user just starts to read/write to arbitrary addresses, chances are that the flash space has some remains from old programs. But the whole idea with the library is that flash space is pre-allocated, and that the user has to stay within this space
First of all, I've used some code I found over at the official Optiboot repo. Using this code I've been able to create a working optiboot.h/cpp library that my new Flash library wraps around. For reference, Optiboot.h/cpp contains the low(ish) flash read-write functions, while Flash.h/cpp contains high-level code that lets you write anything to the flash memory, like strings, structs, and variables. Both libraries will most likely work without modification on the tinyAVR-0/1/2 series too, so feel free to steel this @SpenceKonde!
@technoblogy I was not able to get the
Read_write_without_buffer
example working the megaAVR-0 series though. Do you have any experience with the "modern" AVRs and "write-to-flash" functionality?optiboot.h implements
optiboot_page_erase
,optiboot_page_fill
andoptiboot_page_write
(borrowed from here), but I'm not sure these works differently than on classic AVRS? Here's the example sketch I've not been able to do what I want:Code:
Output:
Another thing. I know these chips have "memory-mapped flash", but I don't know exactly how this works on a very low level. I've not been able to get rid of the
PROGMEM
attribute orpgm_read_byte
, and I'm not sure if this is possible or not. If yes, it will reduce the compiled size of the demo sketches.