MCUdude / MightyCore

Arduino hardware package for ATmega1284, ATmega644, ATmega324, ATmega324PB, ATmega164, ATmega32, ATmega16 and ATmega8535
Other
649 stars 182 forks source link

Questions about Optiboot Flasher #210

Open technoblogy opened 3 years ago

technoblogy commented 3 years ago

I'm planning to use your Optiboot Flasher on the ATmega1284P, but there are a few things I don't understand:

Sorry about the flood of questions!

MCUdude commented 3 years ago

Hi!

I wrote this crude sketch five years ago, so a lot has happened to my coding skills since since then. I haven't really used this functionality myself either, but I'll try to answer your questions as good as I can. Note that this sketch relies on optiboot.h, which is mostly written by @majekw. This is the sketch](https://github.com/majekw/optiboot/blob/spm/optiboot/examples/flash_program/flash_program.ino) SerialReadWrite.ino is based on. As you can see in this sketch, here's where I've copied all the "important things to know" text.

I'm not sure what you're referring to by 'buffer' in the documentation in the header of SerialReadWrite.ino. When you say: Buffer must be page aligned (see declaration of flash_buffer) There is no flash_buffer in the code. I assume this means the block of flash you're writing to, flashSpace[]?

Yes, I'm referring to flashSpace[] here.

In: Writing to EEPROM destroys temporary buffer You can write only once into one location of temporary buffer By 'temporary buffer' do you mean: uint8_t ramBuffer[SPM_PAGESIZE]; and by EEPROM do you mean flash? Why does it get destroyed by writing it to flash?

When looking at the code now, there is no reason why the ramBuffer should be destroyed when writing to EEPROM. ramBuffer is just an array living in RAM, and should not be affected. But it would be great if you could try this just to check if the statement from majekw still is true (for some reason).

Why might you want to do fill-erase-write rather than erase-fill-write?

I am doing erase-fill-write. See code here.

Instead of allocating the page(s) of flash in the middle of PROGMEM as you've done here, could I specify an explicit address for flashSpace? For example, if I understand correctly the bootloader occupies two pages from word addresses 0xFF00 to 0xFFFF. I would like to write to one page of flash just below that, from word addresses 0xFE80 to 0xFEFF. So could I do this?

I've not tried this myself, and I don't have a good answer to your question. However, you can always try this and use Avrdude to dump the flash and look at its content afterward to see if it did work? Perhaps maybe @majekw can give us a straight answer? 🙂

Is it essential to use a RAM buffer, ramBuffer[256], or if I'm short of RAM could I write the data to flash byte by byte, treating it more like an SD card? I realize I would have to provide my own alternative to optiboot_writePage().

As far as I know, you can only write a full page to the flash memory at once. You could of course create a wrapper that treats each page as a byte array, but you'll always have to "write" all the bytes, even though only one has changed.

Finally, I'm puzzled by the code in optiboot_readPage() in optiboot.h: ... Why is it skipping bytes if they are 0 or 255?

The code is based on the read routine from the original sketch. It may be that 0x00 or 0xff represents an empty space? But I do agree that 0 and 255 should be read out as well. If not, it's pretty much useless for anything else than characters.

I can change the optiboot.h and the SerialReadWrite sketch based on what you figure out what works and what don't. Looking forward to seeing what you'll come up with! Is it something that you'll write about on your blog?

technoblogy commented 3 years ago

Thanks for the answers! I'll let you know what I find out after I've done some experiments.

The application I've got in mind is to improve my version of uLisp for the ATmega1284P. Currently it allows you to save the workspace from RAM to the EEPROM on the chip, which limits you to saving 4Kbytes of the 16Kbytes. By saving it to flash instead I could save the entire RAM.

JAndrassy commented 3 years ago

I use optiboot.h in the ArduinoOTA library to store the uploaded sketch binary in upper half of the flash memory. I don't use a memory buffer. The MCU has a buffer for the page. https://github.com/jandrassy/ArduinoOTA/blob/master/src/InternalStorageAVR.cpp

btw: fill-erase-flash is only possible in last 4 flash pages to support advanced bootloader burning

technoblogy commented 3 years ago

@jandrassy Thanks, I'll take a look at your code too.

majekw commented 3 years ago

Lot of questions :-)

  1. flash_buffer is unfortunate name which conflicts with AVR's flash buffer used to store temporary flash page before writing it. 'flash_buffer' variable is just memory in flash where we want to write. That's also a reason for page aligning, because in fact we must write whole pages.
  2. Flash vs. EEPROM note: during writing we must write to internal AVR flash buffer then 'commit' write to real flash location. This internal flash buffer could be erased during operations on EEPROM. It's somehow connected in chip and this limitation is described in AVR documentation.
  3. fill-erase-write vs. erase-fill-write: when you want to preserve some contents of erased page, you don't need RAM to store old contents of erased page in first case. Preparing contents of page in RAM and write all in one go is just more convenient and easier to implement, so second way is more common. There also could be some magic with NRWW/RWW sections, but I don't remember all these quirks right now. For better understanding I recommend to read chip documentation about bootloader, NRWW, RWW sections and self programming.
  4. regarding 'if' with 0 and 255: it was just to output some reasonable characters to serial console, so I was replacing them to dots :-) No other magic here :-)
technoblogy commented 3 years ago

OK, I've got it to work! Here is some feedback. This generally relates to the ATmega1284P and may need changing for the smaller chips.

Finally, here are my demo programs which I've tried to keep as simple as possible.

Writing to an explicit page in far flash

/*
  Test Optiboot Flash on ATmega1284P - explicit page
*/

#include "optiboot.h"

// Bootloader takes from 0x1fc00 to 0x1ffff (byte addresses).
// Write in page immediately below that
uint32_t BaseAddress = 0x1fb00;

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println("Erase");
  optiboot_page_erase(BaseAddress);

  Serial.println("Write data");
  uint32_t addr = BaseAddress;

  // Fill flash buffer. Page is 128 words
  for (unsigned int i=0; i<128; i++) {
    optiboot_page_fill(addr, i); addr++; addr++;
  }

  // Write flash buffer to flash area
  optiboot_page_write(BaseAddress);

  Serial.println("Read data");
  addr = BaseAddress;  
  for (unsigned int i=0; i<128; i++) {
    Serial.print(pgm_read_word_far(addr));
    addr++; addr++; Serial.print(' ');
  }
  Serial.println();

  delay(20000);
}

Writing to a page allocated in PROGMEM

/*
  Test Optiboot Flash on ATmega1284P - allocate page in PROGMEM
*/

#include "optiboot.h"

// Allocate one page of 256 bytes
const uint8_t flashSpace[256] __attribute__ (( aligned(256) )) PROGMEM = { };
uint32_t BaseAddress = (uint32_t)flashSpace;

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println("Erase");
  optiboot_page_erase(BaseAddress);

  Serial.println("Write data");
  uint32_t addr = BaseAddress;

  // Fill flash buffer. Page is 128 words
  for (unsigned int i=0; i<128; i++) {
    optiboot_page_fill(addr, i); addr++; addr++;
  }

  // Write flash buffer to flash area
  optiboot_page_write(BaseAddress);

  Serial.println("Read data");
  addr = BaseAddress;  
  for (unsigned int i=0; i<128; i++) {
    Serial.print(pgm_read_word(addr));
    addr++; addr++; Serial.print(' ');
  }
  Serial.println();

  delay(20000);
}
technoblogy commented 3 years ago

I forgot to say: thanks for providing this really useful feature!

MCUdude commented 3 years ago

@technoblogy thanks for the feedback! I'll go through it later and apply some of the improvements you've suggested.

BTW you can write to the upper part of flash by using the PROGMEM1 attribute:

const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on a page far away"
};

EDIT: Not, that this doesn't work for some reason. Haven't had the time to investigate why. I did change pgm_read_byte_far to pgm_read_byte_far in optiboot_readPage(), but still no success. For reference, PROGMEM1 is defined here.

technoblogy commented 3 years ago

Here's another example, which is what I'm using in uLisp. You simply keep calling FlashWriteInt() to write values to the flash (up to 16Kbytes in this example), and the only other thing you have to do is call FlashEndWrite() at the end. It takes care of the correct sequence of optiboot_page_erase, optiboot_page_fill, and optiboot_page_write calls for you:

const uint32_t BaseAddress = 0x1bc00;

void FlashWriteInt (uint32_t *addr, int data) {
  if (((*addr) & 0xFF) == 0) optiboot_page_erase(BaseAddress + ((*addr) & 0xFF00));
  optiboot_page_fill(BaseAddress + *addr, data);
  if (((*addr) & 0xFF) == 0xFE) optiboot_page_write(BaseAddress + ((*addr) & 0xFF00));
  (*addr)++; (*addr)++;
}

void FlashEndWrite (uint32_t *addr) {
  if (((*addr) & 0xFF) != 0x00) optiboot_page_write((BaseAddress + ((*addr) & 0xFF00)));
}

void loop() {
  uint32_t addr = 0;

  for (int word = 0; word<500; word++) FlashWriteInt(&addr, word);

  FlashEndWrite(&addr);

  for(;;);
}
MCUdude commented 3 years ago

Is there a "proper way" to read content that's stored in "far mem" (64-128kB)? I've tried to modify the optiboot_readPage function, but it seems like the pgm_get_far_address macro requires its parameter to be known at compile time. I'm getting this error:

/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_544907/sketch/SerialReadWrite.ino.cpp.o: In function `optiboot_readPage(unsigned char const*, unsigned char*, unsigned int, char)':
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
/Users/hans/Documents/Arduino/hardware/MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h:137: undefined reference to `r30'
collect2: error: ld returned 1 exit status
Using library Optiboot_flasher at version 1.0
// Function to read a flash page and store it in an array (storage_array[])
void optiboot_readPage(const uint8_t allocated_flash_space[], uint8_t storage_array[], uint16_t page, char blank_character)
{
  uint8_t read_character;
  for(uint16_t j = 0; j < SPM_PAGESIZE; j++) 
  {

    read_character = pgm_read_byte_far(pgm_get_far_address(allocated_flash_space) + j + SPM_PAGESIZE*(page-1));
    if(read_character != 0 && read_character != 255)
      storage_array[j] = read_character; 
    else
      storage_array[j] = blank_character;   
  }
}
JAndrassy commented 3 years ago

RAMZ register

MCUdude commented 3 years ago

The idea is to see if I can create a usable example where I utilize PROGMEM1.

// This array allocates the space you'll be able to write to
static const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one"
};

I want to see if there's possible to read and write to the array when it's located in the far (64kB+) memory space. So far I've been able to read from it by "manually" passing it's memory address, but it's not very elegant. When I try to write to it I either corrupt the bootloader area or the program gets permanently corrupted.

Preferably, this is what I want to acheive:



// This array allocates the space you'll be able to write to
static const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one but still far away!"
};

// ...

// Write content to page 1:
optiboot_writePage(flashSpace, ramBuffer, 1);

// Read page 1 of the allocated space:
optiboot_readPage(flashSpace, ramBuffer, 1); 
// Print page content
Serial.print(F("\nContent of page 1: "));
Serial.println((char*)ramBuffer);
JAndrassy commented 3 years ago

maybe my copy_flash_pages function's source code can help https://github.com/MCUdude/MightyCore/blob/09909376fbf39305e1047bbc2efeb8f67f441620/avr/bootloaders/optiboot_flash/optiboot_flash.c#L1190-L1209

MCUdude commented 3 years ago

Preferably, this is what I want to achieve:

OK, I got it working, but I still can't get around the fact that the address to the array has to be known at compile-time and thus passed to the optiboot_readPage and optiboot_writePage functions. But this isn't really a deal-breaker. I'll continue working on some of the suggestions @technoblogy came up with! There may be breaking changes to optiboot.h, but these will be minor, and absolutely for the best (like getting rid of the 0 and 255 guards in optiboot_read_page for instance).

MCUdude commented 3 years ago

In the SerialReadWrite example I've just updated, you can now easily place all your data in the far memory using PROGMEM1. All you have to do is to add pgm_get_far_address when you're reading or writing:

const uint8_t flashSpace_near[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one, really near!"
};

const uint8_t flashSpace_far[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM1 = {
  "This some default content stored on page one, far, far away..."
};

// write to near mem
optiboot_writePage(flashSpace_near, ramBuffer, pageNumber);

// write to far mem
optiboot_writePage(pgm_get_far_address(flashSpace_far), ramBuffer, pageNumber);

// Read from near mem
optiboot_readPage(flashSpace_near, ramBuffer, page);

// Read from far mem
optiboot_readPage(pgm_get_far_address(flashSpace_far), ramBuffer, page);

I hope these changes to optiboot.h makes it more suitable for real-world applications 🙂 Feel free to bring some feedback! I'd love to further improve it.

technoblogy commented 3 years ago

Personally I find this a bit confusing, and would prefer it in a comment:

"This some default content stored on page one, really near!"

Why would you want to have some default content there? Also, it implies that you're only going to be storing text.

MCUdude commented 3 years ago

It's intended to show what the SerialReadWrite sketch is capable of. In the sketch, you can dump the entire allocated space, and one can see where the text is. As a demo, I think it helps to illustrate what the user is working with. It doesn't make sense in a read-world application, that's for sure.

I'm planning to create other examples as well that show how other data can be handled, with and without a buffer in RAM.

Do you have any ideas on what other examples could be useful to help users utilize the flash read/write feature in their own applications?

technoblogy commented 3 years ago

Perhaps my example?

https://github.com/MCUdude/MightyCore/issues/210#issuecomment-808773024

MCUdude commented 3 years ago

@technoblogy I will include a modified version of your example to show how one can write 16-bit integers straight to the internal buffer. It's very fast and light-weight, but not as user-friendly as I want it to be.

That's why I've created a wrapper library that utilizes Optiboot.h but acts very much like the Arduino EEPROM library! This means that you can easily store all sorts of things like variables, structs, strings, etc, and the library will take care of the low-level stuff. It is template-based, so the compiled size is still very small.

The compiled size without LTO for the example below is 6056 bytes for an ATmega1284P. Approx. 1.7kB is due to printing floats.

Any thoughts? I will publish the library after I've fine-tuned it even more 👍

// Flash_put_get.ino
#include <Flash.h>

struct MyObject
{
  float field1;
  uint8_t field2;
  char name[10];
};

// RAM buffer needed by the Flash library
uint8_t ram_buffer[SPM_PAGESIZE] = {0x00};

// Allocate two flash pages for storing data
#define NUMBER_OF_PAGES 2
const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__((aligned(SPM_PAGESIZE))) PROGMEM = {};

// Flash constructor
Flash flash(flashSpace, sizeof(flashSpace), ram_buffer);

void write_data()
{
  float f = 123.456f;
  uint8_t buffer_address = 0;

  // First, make sure there are no content in out buffer
  flash.clear_buffer();

  // One simple call, with the address first and the object second
  flash.put(buffer_address, f);

  Serial.println(F("Written float data type!"));

  // Data to store
  MyObject customVar =
  {
    3.14f,
    65,
    "MCUdude"
  };

  // Move address to the next byte after float 'f'
  buffer_address += sizeof(float);
  flash.put(buffer_address, customVar);

  // Write buffer to the first allocated flash page (page 0)
  flash.write_page(0);

  // Now let's set a flag on another flash page to indicate that the flash memory contains content
  // Here we're treating the object as an array
  flash.clear_buffer();
  flash[5] = 'X';
  flash.write_page(1);

  Serial.println(F("Written custom data type!\nReset your board to view the contents!\n"));
}

void read_data()
{
  Serial.println(F("Read float from flash: "));

  // fetch first flash page
  flash.fetch_page(0);

  float f = 0.00f; // Variable to store data read from flash
  uint8_t buffer_address = 0; // Buffer address to start from

  // Get the float data from flash at position 'buffer_address'
  flash.get(buffer_address, f);
  Serial.print(F("The value of f is now: "));
  Serial.println(f, 3);

  buffer_address += sizeof(float); // Move address to the next byte after float 'f'
  MyObject customVar; // Variable to store custom object read from flash.

  flash.get(buffer_address, customVar);

  Serial.println(F("Read custom object from flash: "));
  Serial.println(customVar.field1);
  Serial.println(customVar.field2);
  Serial.println(customVar.name);
}

void setup()
{
  delay(2000);
  Serial.begin(9600);

  // Fetch flash page 1, where we may have a flag
  flash.fetch_page(1);

  // Check if our flag is present
  if(flash[5] == 'X')
  {
    Serial.println(F("Content found! Content:"));
    read_data();
  }
  else
  {
    Serial.println(F("No content found! Writing new content..."));
    write_data();
  }
}

void loop()
{
}

Serial monitor output after upload:

No content found! Writing new content...
Written float data type!
Written custom data type!
Reset your board to view the contents!

(I press the reset button)

Content found! Content:
Read float from flash: 
The value of f is now: 123.456
Read custom object from flash: 
3.14
65
MCUdude
technoblogy commented 3 years ago

I'm not sure why you've provided:

uint8_t ram_buffer[SPM_PAGESIZE] = {0x00};

because it suggests that you need a RAM buffer to use the Optiboot Flash, which isn't the case; a point I made in my comments at:

https://github.com/MCUdude/MightyCore/issues/210#issuecomment-808702140

MCUdude commented 3 years ago

One does not need a RAM buffer to read/write to flash, but it makes it much more convenient to have a pool you can read and write to, just like I've shown in the example.

If you have lots of structs and variables scattered around on a page, how would you work with it if you didn't have a buffer you can dump the content into before parsing the data? You may be able to reduce the RAM usage a little, but I bet the code is not going to be as pretty.

Optiboot.h does not require a RAM buffer, and I'm going to create an example showing this Flash.h will most likely require a RAM buffer because It's supposed to be easy to use and act much like the EEPROM library. I'd love to get rid of the buffer, but I don't really see how that could be accomplished. Is it possible to read from the internal flash buffer, so this could be used as a "pool"?

technoblogy commented 3 years ago

Fair enough.

Is it possible to read from the internal flash buffer, so this could be used as a "pool"?

I'm not sure.

technoblogy commented 3 years ago

Do you think it will be possible to implement this (from my earlier comments):

So I think it might be useful to provide a function like the one in Spence Konde's DxCore called Flash.checkWritable() which checks for obvious things. I'm not sure how to write this.

It's pretty important, because currently if you try to use Optiboot.h with the wrong bootloader it just crashes your program in a major way. It should at least check that you've got the correct bootloader.

MCUdude commented 3 years ago

How about this?

https://github.com/MCUdude/MightyCore/blob/e12eda5cd7f82acdb5eca1d8db3eb1bc97afb437/avr/libraries/Optiboot_flasher/src/Flash.cpp#L51-L86

technoblogy commented 3 years ago

Yes, that looks perfect.

The only concern I have is that, in order to use Flash::check_writable(), I have to create a Flash instance which appears to carry an overhead of 10 bytes, which I don't need. I would prefer it not to be a C++ routine.

I assume you're leaving optiboot.h and optiboot.cpp unchanged?

MCUdude commented 3 years ago

I was already thinking about this. See my latest commit.

https://github.com/MCUdude/MightyCore/blob/5b1dac881e4fa4979e4e5c7f12cf457ce8e93ef1/avr/libraries/Optiboot_flasher/src/optiboot.cpp#L30-L65

BTW the reason why I created optiboot.cpp was because just having a header file made it a pain to include in other files. I constantly got "multiple definitions of..." errors. I also went with cpp since a few functions are overloaded, which is not supported in C.

technoblogy commented 3 years ago

OK, great. Now I can call optiboot_check_writable(). I assume optiboot_page_erase(), optiboot_page_fill(), and optiboot_page_write() haven't changed?

MCUdude commented 3 years ago

OK, great. Now I can call optiboot_check_writable(). I assume optiboot_page_erase(), optiboot_page_fill(), and optiboot_page_write() haven't changed?

Correct! Nothing has changed in these low-level functions, apart from me adding more comments in the code.

MCUdude commented 3 years ago

@technoblogy this is the example I'm going to provide that's based on your example code. It isn't identical at all, but it borrows your principles. I'm also demonstrating how to write 8-bit data to flash:

/***********************************************************************|
| Optiboot Flash read/write                                             |
|                                                                       |
| Read_write_without_buffer.ino                                         |
|                                                                       |
| A library for interfacing with Optiboot Flash's write functionality   |
| Developed in 2021 by MCUdude                                          |
| https://github.com/MCUdude/                                           |
|                                                                       |
| In this low-level example we write 16-bit values to one flash page,   |
| and 8-bit values to another page. What's different about this         |
| example is that we do this without using a RAM buffer where the       |
| contents are stored before writing to flash. Instead, the internal,   |
| temporary buffer is used. By doing this we reduce RAM usage, but it   |
| is not nearly as user friendly as using the Flash library.            |
|                                                                       |
| IMPORTANT THINGS:                                                     |
| - All flash content gets erased after each upload cycle               |
| - Allocated flash space must be page aligned (it is in this example)  |
| - Writing to EEPROM destroys temporary buffer so make sure you call   |
|   optiboot_page_write before reusing the buffer or using EEPROM       |
| - You can write only once into one location of temporary buffer       |
|***********************************************************************/

#include <optiboot.h>

// Workaround for devices that has 64kiB flash or less
#ifndef pgm_read_byte_far
#define pgm_read_byte_far pgm_read_byte_near
#endif
#ifndef pgm_get_far_address
#define pgm_get_far_address
#endif

// Allocate one flash pages for storing data. If you want to allocate space in the high progmem (>64kiB)
// You can replace PROGMEM with PROGMEM1
#define NUMBER_OF_PAGES 2
const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__((aligned(SPM_PAGESIZE))) PROGMEM = {};

// Function for writing 16-bit integers to a flash page
void flash_write_int(uint32_t base_addr, uint16_t offset_addr, int16_t data)
{
  // Start by erasing the flash page before we start writing to the buffer
  if ((offset_addr & 0xFF) == 0)
    optiboot_page_erase(base_addr + (offset_addr & 0xFF00));

  // Write the 16-bit value to the buffer
  optiboot_page_fill(base_addr + offset_addr, data);

  // Write the buffer to flash when the buffer is full
  if ((offset_addr & 0xFF) == 0xFE)
    optiboot_page_write(base_addr + (offset_addr & 0xFF00));
}

// Function to write bytes to a flash page
void flash_write_byte(uint32_t base_addr, uint16_t offset_addr, uint8_t data)
{
  static uint8_t low_byte = 0;

  if ((offset_addr & 0xFF) == 0)
    optiboot_page_erase(base_addr + (offset_addr & 0xFF00));

  if (!(offset_addr & 0x01)) // Address is 2, 4, 6 etc.
    low_byte = data;
  else                       // Address is 1, 3, 5 etc.
    optiboot_page_fill(base_addr + offset_addr, (data << 8) | low_byte);

  if ((offset_addr & 0xFF) == 0xFE)
    optiboot_page_write(base_addr + (offset_addr & 0xFF00));
}

// Function to force a flash page write operation
void flash_end_write(uint32_t base_addr, uint16_t offset_addr)
{
  // Write the buffer to flash if there are any contents in the buffer
  if ((offset_addr & 0xFF) != 0x00)
    optiboot_page_write(base_addr + (offset_addr & 0xFF00));
}

void setup()
{
  delay(2000);
  Serial.begin(9600);

  static uint16_t addr = 0;

  Serial.print(F("Filling up flash page 0 with 16-bit values...\n"));
  // Fill the first flash page (page 0) with 16-bit values (0x100 to 0x01FF)
  for(uint8_t data = 0; data < (SPM_PAGESIZE / 2); data++)
  { 
    flash_write_int(pgm_get_far_address(flashSpace), addr, data + 0x0100); // Write data
    addr += 2; // Increase memory address by two since we're writing 16-bit values
  }
  // Force an end write in case it hasn't already been done in flash_write_int
  flash_end_write(pgm_get_far_address(flashSpace), --addr);

  Serial.print(F("Filling up flash page 1 with 8-bit values...\n"));
  // Fill the second flash page (page 1) with 0-bit values (0x00 to 0x0FF)
  for(uint16_t data = 0; data < SPM_PAGESIZE; data++)
  {
    addr++; // Increase memory address by one since we're writing 8-bit values
    flash_write_byte(pgm_get_far_address(flashSpace), addr, data); // Write data
  }
  // Force an end write in case it hasn't already been done in flash_write_byte
  flash_end_write(pgm_get_far_address(flashSpace), addr);

  Serial.print(F("Flash pages filled. Reading back their content.\nPage 0:\n"));
  for(uint16_t i = 0; i < SPM_PAGESIZE; i += 2)
    Serial.printf(F("Flash mem addr: 0x%05lx, content: 0x%04x\n"), pgm_get_far_address(flashSpace) + i, (pgm_read_byte_far(pgm_get_far_address(flashSpace) + i + 1) << 8) + (pgm_read_byte_far(pgm_get_far_address(flashSpace) + i)));

  Serial.println(F("\n\nPage 1:"));
  for(uint16_t i = 0; i < SPM_PAGESIZE; i++)
    Serial.printf(F("Flash mem addr: 0x%05lx, content: 0x%02x\n"), pgm_get_far_address(flashSpace) + SPM_PAGESIZE + i, pgm_read_byte_far(pgm_get_far_address(flashSpace) + SPM_PAGESIZE + i));
}

void loop()
{

}
technoblogy commented 3 years ago

I think worth mentioning in the comment that you must call flash_end_write() when you've finished, otherwise you may end up with the last page not written.

MCUdude commented 3 years ago

I think worth mentioning in the comment that you must call flash_end_write() when you've finished, otherwise you may end up with the last page not written.

Thanks! I'll point this out

MCUdude commented 3 years ago

For some reason, the example sketch I just posted doesn't work with devices that has a flash page size of 128 bytes or less, which is all chips with 32kiB flash or less. It does compile just fine, but the data isn't written to flash properly. I guess I have a late night ahead of me...

JAndrassy commented 3 years ago

For some reason, the example sketch I just posted doesn't work with devices that has a flash page size of 128 bytes or less, which is all chips with 32kiB flash or less. It does compile just fine, but the data isn't written to flash properly. I guess I have a late night ahead of me...

don't use _far_ for those devices.

MCUdude commented 3 years ago

don't use far for those devices.

_far_ isn't even defined for these devices, that's why I added:

#ifndef pgm_read_byte_far
#define pgm_read_byte_far pgm_read_byte_near
#endif
#ifndef pgm_get_far_address
#define pgm_get_far_address
#endif

...at the beginning of the sketch. And the code works fine on an ATmega644P, which only has "near mem", but has 256-byte flash pages.

technoblogy commented 3 years ago

I think it might be the tests for the first and last word in the page:

if ((offset_addr & 0xFF) == 0)

if ((offset_addr & 0xFF) == 0xFE)

These assume the page size is 0xFF. For a page size of 0x7F the second test should be:

if ((offset_addr & 0xFF) == 0x7E)

Also, the test in the flash_write_byte() function for a page size of 0xFF should be:

if ((offset_addr & 0xFF) == 0xFF)

and for a page size of 0x7F it should be:

if ((offset_addr & 0xFF) == 0x7F)
MCUdude commented 3 years ago

@technoblogy thanks for the input, but for some reason, I'm still getting garbage in the Serial monitor.

In the first place, I'm writing 16-bit integers starting from 0x0100 to 0x17F. The output looks like this:

Page 0:
Flash mem addr: 0x00180, content: 0x0100
Flash mem addr: 0x00182, content: 0x0302
Flash mem addr: 0x00184, content: 0x0504
Flash mem addr: 0x00186, content: 0x0706
Flash mem addr: 0x00188, content: 0x0908
Flash mem addr: 0x0018a, content: 0x0b0a
Flash mem addr: 0x0018c, content: 0x0d0c
Flash mem addr: 0x0018e, content: 0x0f0e
Flash mem addr: 0x00190, content: 0x1110
Flash mem addr: 0x00192, content: 0x1312
Flash mem addr: 0x00194, content: 0x1514
...
...
Flash mem addr: 0x001fc, content: 0x7d7c
Flash mem addr: 0x001fe, content: 0x7f7e

For the second page I'm writing 8-bit values from 0x00 to 0x7F. This is the output:

Page1:
Flash mem addr: 0x00200, content: 0x00
Flash mem addr: 0x00201, content: 0x00
Flash mem addr: 0x00202, content: 0x00
Flash mem addr: 0x00203, content: 0x00
Flash mem addr: 0x00204, content: 0x00
Flash mem addr: 0x00205, content: 0x00
Flash mem addr: 0x00206, content: 0x00
Flash mem addr: 0x00207, content: 0x00
...
...
Flash mem addr: 0x0027e, content: 0x00
Flash mem addr: 0x0027f, content: 0x00
technoblogy commented 3 years ago

What chip (with what page size) are you using?

MCUdude commented 3 years ago

This test was done on an ATmega16 with a page size of 128 bytes. But I have tried with an ATmega324P (128 bytes page size), and I'm getting the exact same output. But I have every MightyCore compatible chip in my drawer if that could help...

technoblogy commented 3 years ago

Are there conditions under which it does work?

tomaskovacik commented 3 years ago

dump whole flash with avrdude and check if it is problem with writing or reading

MCUdude commented 3 years ago

Are there conditions under which it does work?

I have not been able to make it work on any chip with 64 or 128 bytes page size.

dump whole flash with avrdude and check if it is problem with writing or reading

Here are the relevant parts of the flash dump:

:200160002D6269742076616C7565732E2E2E0A0000000000000000000000000000000000CF
:20018000000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F6F (Addr: 0x0180 - first page)
:2001A000202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F4F
:2001C000404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F2F
:2001E000606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F0F
:200200000000000000000000000000000000000000000000000000000000000000000000DE (Addr: 0x0200 - second page)
:200220000000000000000000000000000000000000000000000000000000000000000000BE
:2002400000000000000000000000000000000000000000000000000000000000000000009E
:2002600000000000000000000000000000000000000000000000000000000000000000007E
:20028000700511241FBECFEFD8E0DEBFCDBF11E0A0E0B1E0E2E9F2E102C005900D92A631CB
technoblogy commented 3 years ago

Now I look at it again I realise that my example needs changing quite a bit for page sizes other than 256.

 optiboot_page_erase(base_addr + (offset_addr & 0xFF00));

and

optiboot_page_write(base_addr + (offset_addr & 0xFF00));

should be changed to this for 128-byte pages:

 optiboot_page_erase(base_addr + (offset_addr & 0xFF80));

and:

optiboot_page_write(base_addr + (offset_addr & 0xFF80));
MCUdude commented 3 years ago

Well, that certainly did the trick, thank you @technoblogy! I've tested it on an ATmega8535 (64 byte page), ATmega324P (128 byte page), ATmega644P (256 byte page) and ATmega1284 (256 byte page). I can also confirm that it works excellent in high progmem too. It can easilly be tested by defining the flash space in PROGMEM1 rather than PROGMEM.

``` Filling up flash page 0 with 16-bit values... Filling up flash page 1 with 8-bit values... Flash pages filled. Reading back their content. Page 0: Flash mem addr: 0x00180, content: 0x0100 Flash mem addr: 0x00182, content: 0x0101 Flash mem addr: 0x00184, content: 0x0102 Flash mem addr: 0x00186, content: 0x0103 Flash mem addr: 0x00188, content: 0x0104 Flash mem addr: 0x0018a, content: 0x0105 Flash mem addr: 0x0018c, content: 0x0106 Flash mem addr: 0x0018e, content: 0x0107 Flash mem addr: 0x00190, content: 0x0108 Flash mem addr: 0x00192, content: 0x0109 Flash mem addr: 0x00194, content: 0x010a Flash mem addr: 0x00196, content: 0x010b Flash mem addr: 0x00198, content: 0x010c Flash mem addr: 0x0019a, content: 0x010d Flash mem addr: 0x0019c, content: 0x010e Flash mem addr: 0x0019e, content: 0x010f Flash mem addr: 0x001a0, content: 0x0110 Flash mem addr: 0x001a2, content: 0x0111 Flash mem addr: 0x001a4, content: 0x0112 Flash mem addr: 0x001a6, content: 0x0113 Flash mem addr: 0x001a8, content: 0x0114 Flash mem addr: 0x001aa, content: 0x0115 Flash mem addr: 0x001ac, content: 0x0116 Flash mem addr: 0x001ae, content: 0x0117 Flash mem addr: 0x001b0, content: 0x0118 Flash mem addr: 0x001b2, content: 0x0119 Flash mem addr: 0x001b4, content: 0x011a Flash mem addr: 0x001b6, content: 0x011b Flash mem addr: 0x001b8, content: 0x011c Flash mem addr: 0x001ba, content: 0x011d Flash mem addr: 0x001bc, content: 0x011e Flash mem addr: 0x001be, content: 0x011f Flash mem addr: 0x001c0, content: 0x0120 Flash mem addr: 0x001c2, content: 0x0121 Flash mem addr: 0x001c4, content: 0x0122 Flash mem addr: 0x001c6, content: 0x0123 Flash mem addr: 0x001c8, content: 0x0124 Flash mem addr: 0x001ca, content: 0x0125 Flash mem addr: 0x001cc, content: 0x0126 Flash mem addr: 0x001ce, content: 0x0127 Flash mem addr: 0x001d0, content: 0x0128 Flash mem addr: 0x001d2, content: 0x0129 Flash mem addr: 0x001d4, content: 0x012a Flash mem addr: 0x001d6, content: 0x012b Flash mem addr: 0x001d8, content: 0x012c Flash mem addr: 0x001da, content: 0x012d Flash mem addr: 0x001dc, content: 0x012e Flash mem addr: 0x001de, content: 0x012f Flash mem addr: 0x001e0, content: 0x0130 Flash mem addr: 0x001e2, content: 0x0131 Flash mem addr: 0x001e4, content: 0x0132 Flash mem addr: 0x001e6, content: 0x0133 Flash mem addr: 0x001e8, content: 0x0134 Flash mem addr: 0x001ea, content: 0x0135 Flash mem addr: 0x001ec, content: 0x0136 Flash mem addr: 0x001ee, content: 0x0137 Flash mem addr: 0x001f0, content: 0x0138 Flash mem addr: 0x001f2, content: 0x0139 Flash mem addr: 0x001f4, content: 0x013a Flash mem addr: 0x001f6, content: 0x013b Flash mem addr: 0x001f8, content: 0x013c Flash mem addr: 0x001fa, content: 0x013d Flash mem addr: 0x001fc, content: 0x013e Flash mem addr: 0x001fe, content: 0x013f Page 1: Flash mem addr: 0x00200, content: 0x00 Flash mem addr: 0x00201, content: 0x01 Flash mem addr: 0x00202, content: 0x02 Flash mem addr: 0x00203, content: 0x03 Flash mem addr: 0x00204, content: 0x04 Flash mem addr: 0x00205, content: 0x05 Flash mem addr: 0x00206, content: 0x06 Flash mem addr: 0x00207, content: 0x07 Flash mem addr: 0x00208, content: 0x08 Flash mem addr: 0x00209, content: 0x09 Flash mem addr: 0x0020a, content: 0x0a Flash mem addr: 0x0020b, content: 0x0b Flash mem addr: 0x0020c, content: 0x0c Flash mem addr: 0x0020d, content: 0x0d Flash mem addr: 0x0020e, content: 0x0e Flash mem addr: 0x0020f, content: 0x0f Flash mem addr: 0x00210, content: 0x10 Flash mem addr: 0x00211, content: 0x11 Flash mem addr: 0x00212, content: 0x12 Flash mem addr: 0x00213, content: 0x13 Flash mem addr: 0x00214, content: 0x14 Flash mem addr: 0x00215, content: 0x15 Flash mem addr: 0x00216, content: 0x16 Flash mem addr: 0x00217, content: 0x17 Flash mem addr: 0x00218, content: 0x18 Flash mem addr: 0x00219, content: 0x19 Flash mem addr: 0x0021a, content: 0x1a Flash mem addr: 0x0021b, content: 0x1b Flash mem addr: 0x0021c, content: 0x1c Flash mem addr: 0x0021d, content: 0x1d Flash mem addr: 0x0021e, content: 0x1e Flash mem addr: 0x0021f, content: 0x1f Flash mem addr: 0x00220, content: 0x20 Flash mem addr: 0x00221, content: 0x21 Flash mem addr: 0x00222, content: 0x22 Flash mem addr: 0x00223, content: 0x23 Flash mem addr: 0x00224, content: 0x24 Flash mem addr: 0x00225, content: 0x25 Flash mem addr: 0x00226, content: 0x26 Flash mem addr: 0x00227, content: 0x27 Flash mem addr: 0x00228, content: 0x28 Flash mem addr: 0x00229, content: 0x29 Flash mem addr: 0x0022a, content: 0x2a Flash mem addr: 0x0022b, content: 0x2b Flash mem addr: 0x0022c, content: 0x2c Flash mem addr: 0x0022d, content: 0x2d Flash mem addr: 0x0022e, content: 0x2e Flash mem addr: 0x0022f, content: 0x2f Flash mem addr: 0x00230, content: 0x30 Flash mem addr: 0x00231, content: 0x31 Flash mem addr: 0x00232, content: 0x32 Flash mem addr: 0x00233, content: 0x33 Flash mem addr: 0x00234, content: 0x34 Flash mem addr: 0x00235, content: 0x35 Flash mem addr: 0x00236, content: 0x36 Flash mem addr: 0x00237, content: 0x37 Flash mem addr: 0x00238, content: 0x38 Flash mem addr: 0x00239, content: 0x39 Flash mem addr: 0x0023a, content: 0x3a Flash mem addr: 0x0023b, content: 0x3b Flash mem addr: 0x0023c, content: 0x3c Flash mem addr: 0x0023d, content: 0x3d Flash mem addr: 0x0023e, content: 0x3e Flash mem addr: 0x0023f, content: 0x3f Flash mem addr: 0x00240, content: 0x40 Flash mem addr: 0x00241, content: 0x41 Flash mem addr: 0x00242, content: 0x42 Flash mem addr: 0x00243, content: 0x43 Flash mem addr: 0x00244, content: 0x44 Flash mem addr: 0x00245, content: 0x45 Flash mem addr: 0x00246, content: 0x46 Flash mem addr: 0x00247, content: 0x47 Flash mem addr: 0x00248, content: 0x48 Flash mem addr: 0x00249, content: 0x49 Flash mem addr: 0x0024a, content: 0x4a Flash mem addr: 0x0024b, content: 0x4b Flash mem addr: 0x0024c, content: 0x4c Flash mem addr: 0x0024d, content: 0x4d Flash mem addr: 0x0024e, content: 0x4e Flash mem addr: 0x0024f, content: 0x4f Flash mem addr: 0x00250, content: 0x50 Flash mem addr: 0x00251, content: 0x51 Flash mem addr: 0x00252, content: 0x52 Flash mem addr: 0x00253, content: 0x53 Flash mem addr: 0x00254, content: 0x54 Flash mem addr: 0x00255, content: 0x55 Flash mem addr: 0x00256, content: 0x56 Flash mem addr: 0x00257, content: 0x57 Flash mem addr: 0x00258, content: 0x58 Flash mem addr: 0x00259, content: 0x59 Flash mem addr: 0x0025a, content: 0x5a Flash mem addr: 0x0025b, content: 0x5b Flash mem addr: 0x0025c, content: 0x5c Flash mem addr: 0x0025d, content: 0x5d Flash mem addr: 0x0025e, content: 0x5e Flash mem addr: 0x0025f, content: 0x5f Flash mem addr: 0x00260, content: 0x60 Flash mem addr: 0x00261, content: 0x61 Flash mem addr: 0x00262, content: 0x62 Flash mem addr: 0x00263, content: 0x63 Flash mem addr: 0x00264, content: 0x64 Flash mem addr: 0x00265, content: 0x65 Flash mem addr: 0x00266, content: 0x66 Flash mem addr: 0x00267, content: 0x67 Flash mem addr: 0x00268, content: 0x68 Flash mem addr: 0x00269, content: 0x69 Flash mem addr: 0x0026a, content: 0x6a Flash mem addr: 0x0026b, content: 0x6b Flash mem addr: 0x0026c, content: 0x6c Flash mem addr: 0x0026d, content: 0x6d Flash mem addr: 0x0026e, content: 0x6e Flash mem addr: 0x0026f, content: 0x6f Flash mem addr: 0x00270, content: 0x70 Flash mem addr: 0x00271, content: 0x71 Flash mem addr: 0x00272, content: 0x72 Flash mem addr: 0x00273, content: 0x73 Flash mem addr: 0x00274, content: 0x74 Flash mem addr: 0x00275, content: 0x75 Flash mem addr: 0x00276, content: 0x76 Flash mem addr: 0x00277, content: 0x77 Flash mem addr: 0x00278, content: 0x78 Flash mem addr: 0x00279, content: 0x79 Flash mem addr: 0x0027a, content: 0x7a Flash mem addr: 0x0027b, content: 0x7b Flash mem addr: 0x0027c, content: 0x7c Flash mem addr: 0x0027d, content: 0x7d Flash mem addr: 0x0027e, content: 0x7e Flash mem addr: 0x0027f, content: 0x7f ```

On the other side, the code now looks more cryptic than ever. But at least these functions will work on pretty much any classic AVR without the need for modification:

// Function for writing 16-bit integers to a flash page
void flash_write_int(uint32_t base_addr, uint16_t offset_addr, int16_t data)
{
  // Start by erasing the flash page before we start writing to the buffer
  if ((offset_addr & (SPM_PAGESIZE - 1)) == 0)
    optiboot_page_erase(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));

  // Write the 16-bit value to the buffer
  optiboot_page_fill(base_addr + offset_addr, data);

  // Write the buffer to flash when the buffer is full
  if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2))
    optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));
}

// Function to write bytes to a flash page
void flash_write_byte(uint32_t base_addr, uint16_t offset_addr, uint8_t data)
{
  static uint8_t low_byte = 0;

  if ((offset_addr & (SPM_PAGESIZE - 1)) == 0)
    optiboot_page_erase(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));

  if (!(offset_addr & 0x01)) // Address is 2, 4, 6 etc.
    low_byte = data;
  else                       // Address is 1, 3, 5 etc.
    optiboot_page_fill(base_addr + offset_addr, (data << 8) | low_byte);

  if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2))
    optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));
}

// Function to force a flash page write operation
void flash_end_write(uint32_t base_addr, uint16_t offset_addr)
{
  // Write the buffer to flash if there are any contents in the buffer
  if ((offset_addr & 0xFF) != 0x00)
    optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));
}
technoblogy commented 3 years ago

Great. Glad it works, and sorry it took me a couple of attempts to work out what the problem was!

Just one thing that people should be warned about: I'm not sure what would happen if flashSpace[] happens to overlap from near memory to far memory. Do you think that would work?

MCUdude commented 3 years ago

Great. Glad it works, and sorry it took me a couple of attempts to work out what the problem was!

No worries. To be honest I haven't been working on this the entire evening, we've also had some delicious easter lamb tonight that took a while to prepare.

Just one thing that people should be warned about: I'm not sure what would happen if flashSpace[] happens to overlap from near memory to far memory. Do you think that would work?

I'm not sure that could even happen as long as they work with a pre-allocated space and not just an arbitrary start address. If you stick with PROGMEM you can only allocate what's left in the near mem. If you use an AVR with 256kiB flash and you're using PROGMEM1, you can still only allocate an array with 0xFFFF places.

I modified the sketch to take a memory address instead, 0xFF00, and filled two pages from this address. This means that the first page will be in low mem, and the other one in high mem. The only "real" modification I had to do apart from commenting out the allocated flash space array and creating a constant with the same name that contains the start address, I had to remove pgm_get_far_address becuase it would fetch the pointer to the memory address I provided, not the address itself.

Anyways, here's the sketch and output if you're interested:

Sketch:

```c++ #include // Workaround for devices that has 64kiB flash or less #ifndef pgm_read_byte_far #define pgm_read_byte_far pgm_read_byte_near #endif // "disable" pgm_get_far_address #undef pgm_get_far_address #define pgm_get_far_address const uint32_t flashSpace = 0xFF00; //#define NUMBER_OF_PAGES 2 //const uint8_t flashSpace[SPM_PAGESIZE * NUMBER_OF_PAGES] __attribute__((aligned(SPM_PAGESIZE))) PROGMEM = {}; // Function for writing 16-bit integers to a flash page void flash_write_int(uint32_t base_addr, uint16_t offset_addr, int16_t data) { // Start by erasing the flash page before we start writing to the buffer if ((offset_addr & (SPM_PAGESIZE - 1)) == 0) optiboot_page_erase(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); // Write the 16-bit value to the buffer optiboot_page_fill(base_addr + offset_addr, data); // Write the buffer to flash when the buffer is full if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2)) optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } // Function to write bytes to a flash page void flash_write_byte(uint32_t base_addr, uint16_t offset_addr, uint8_t data) { static uint8_t low_byte = 0; if ((offset_addr & (SPM_PAGESIZE - 1)) == 0) optiboot_page_erase(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); if (!(offset_addr & 0x01)) // Address is 2, 4, 6 etc. low_byte = data; else // Address is 1, 3, 5 etc. optiboot_page_fill(base_addr + offset_addr, (data << 8) | low_byte); if ((offset_addr & 0xFF) == (SPM_PAGESIZE - 2)) optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } // Function to force a flash page write operation void flash_end_write(uint32_t base_addr, uint16_t offset_addr) { // Write the buffer to flash if there are any contents in the buffer if ((offset_addr & 0xFF) != 0x00) optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1))); } void setup() { delay(2000); Serial.begin(9600); static uint16_t addr = 0; Serial.print(F("Filling up flash page 0 with 16-bit values...\n")); // Fill the first flash page (page 0) with 16-bit values (0x100 to 0x01FF) for(uint8_t data = 0; data < (SPM_PAGESIZE / 2); data++) { flash_write_int(pgm_get_far_address(flashSpace), addr, data + 0x0100); // Write data addr += 2; // Increase memory address by two since we're writing 16-bit values } // Force an end write in case it hasn't already been done in flash_write_int flash_end_write(pgm_get_far_address(flashSpace), --addr); Serial.print(F("Filling up flash page 1 with 8-bit values...\n")); // Fill the second flash page (page 1) with 0-bit values (0x00 to 0x0FF) for(uint16_t data = 0; data < SPM_PAGESIZE; data++) { addr++; // Increase memory address by one since we're writing 8-bit values flash_write_byte(pgm_get_far_address(flashSpace), addr, data); // Write data } // Force an end write in case it hasn't already been done in flash_write_byte flash_end_write(pgm_get_far_address(flashSpace), addr); Serial.print(F("Flash pages filled. Reading back their content.\nPage 0:\n")); for(uint16_t i = 0; i < SPM_PAGESIZE; i += 2) Serial.printf(F("Flash mem addr: 0x%05lx, content: 0x%04x\n"), pgm_get_far_address(flashSpace) + i, (pgm_read_byte_far(pgm_get_far_address(flashSpace) + i + 1) << 8) + (pgm_read_byte_far(pgm_get_far_address(flashSpace) + i))); Serial.println(F("\n\nPage 1:")); for(uint16_t i = 0; i < SPM_PAGESIZE; i++) Serial.printf(F("Flash mem addr: 0x%05lx, content: 0x%02x\n"), pgm_get_far_address(flashSpace) + SPM_PAGESIZE + i, pgm_read_byte_far(pgm_get_far_address(flashSpace) + SPM_PAGESIZE + i)); } void loop() { } ```

Output:

``` Filling up flash page 0 with 16-bit values... Filling up flash page 1 with 8-bit values... Flash pages filled. Reading back their content. Page 0: Flash mem addr: 0x0ff00, content: 0x0100 Flash mem addr: 0x0ff02, content: 0x0101 Flash mem addr: 0x0ff04, content: 0x0102 Flash mem addr: 0x0ff06, content: 0x0103 Flash mem addr: 0x0ff08, content: 0x0104 Flash mem addr: 0x0ff0a, content: 0x0105 Flash mem addr: 0x0ff0c, content: 0x0106 Flash mem addr: 0x0ff0e, content: 0x0107 Flash mem addr: 0x0ff10, content: 0x0108 Flash mem addr: 0x0ff12, content: 0x0109 Flash mem addr: 0x0ff14, content: 0x010a Flash mem addr: 0x0ff16, content: 0x010b Flash mem addr: 0x0ff18, content: 0x010c Flash mem addr: 0x0ff1a, content: 0x010d Flash mem addr: 0x0ff1c, content: 0x010e Flash mem addr: 0x0ff1e, content: 0x010f Flash mem addr: 0x0ff20, content: 0x0110 Flash mem addr: 0x0ff22, content: 0x0111 Flash mem addr: 0x0ff24, content: 0x0112 Flash mem addr: 0x0ff26, content: 0x0113 Flash mem addr: 0x0ff28, content: 0x0114 Flash mem addr: 0x0ff2a, content: 0x0115 Flash mem addr: 0x0ff2c, content: 0x0116 Flash mem addr: 0x0ff2e, content: 0x0117 Flash mem addr: 0x0ff30, content: 0x0118 Flash mem addr: 0x0ff32, content: 0x0119 Flash mem addr: 0x0ff34, content: 0x011a Flash mem addr: 0x0ff36, content: 0x011b Flash mem addr: 0x0ff38, content: 0x011c Flash mem addr: 0x0ff3a, content: 0x011d Flash mem addr: 0x0ff3c, content: 0x011e Flash mem addr: 0x0ff3e, content: 0x011f Flash mem addr: 0x0ff40, content: 0x0120 Flash mem addr: 0x0ff42, content: 0x0121 Flash mem addr: 0x0ff44, content: 0x0122 Flash mem addr: 0x0ff46, content: 0x0123 Flash mem addr: 0x0ff48, content: 0x0124 Flash mem addr: 0x0ff4a, content: 0x0125 Flash mem addr: 0x0ff4c, content: 0x0126 Flash mem addr: 0x0ff4e, content: 0x0127 Flash mem addr: 0x0ff50, content: 0x0128 Flash mem addr: 0x0ff52, content: 0x0129 Flash mem addr: 0x0ff54, content: 0x012a Flash mem addr: 0x0ff56, content: 0x012b Flash mem addr: 0x0ff58, content: 0x012c Flash mem addr: 0x0ff5a, content: 0x012d Flash mem addr: 0x0ff5c, content: 0x012e Flash mem addr: 0x0ff5e, content: 0x012f Flash mem addr: 0x0ff60, content: 0x0130 Flash mem addr: 0x0ff62, content: 0x0131 Flash mem addr: 0x0ff64, content: 0x0132 Flash mem addr: 0x0ff66, content: 0x0133 Flash mem addr: 0x0ff68, content: 0x0134 Flash mem addr: 0x0ff6a, content: 0x0135 Flash mem addr: 0x0ff6c, content: 0x0136 Flash mem addr: 0x0ff6e, content: 0x0137 Flash mem addr: 0x0ff70, content: 0x0138 Flash mem addr: 0x0ff72, content: 0x0139 Flash mem addr: 0x0ff74, content: 0x013a Flash mem addr: 0x0ff76, content: 0x013b Flash mem addr: 0x0ff78, content: 0x013c Flash mem addr: 0x0ff7a, content: 0x013d Flash mem addr: 0x0ff7c, content: 0x013e Flash mem addr: 0x0ff7e, content: 0x013f Flash mem addr: 0x0ff80, content: 0x0140 Flash mem addr: 0x0ff82, content: 0x0141 Flash mem addr: 0x0ff84, content: 0x0142 Flash mem addr: 0x0ff86, content: 0x0143 Flash mem addr: 0x0ff88, content: 0x0144 Flash mem addr: 0x0ff8a, content: 0x0145 Flash mem addr: 0x0ff8c, content: 0x0146 Flash mem addr: 0x0ff8e, content: 0x0147 Flash mem addr: 0x0ff90, content: 0x0148 Flash mem addr: 0x0ff92, content: 0x0149 Flash mem addr: 0x0ff94, content: 0x014a Flash mem addr: 0x0ff96, content: 0x014b Flash mem addr: 0x0ff98, content: 0x014c Flash mem addr: 0x0ff9a, content: 0x014d Flash mem addr: 0x0ff9c, content: 0x014e Flash mem addr: 0x0ff9e, content: 0x014f Flash mem addr: 0x0ffa0, content: 0x0150 Flash mem addr: 0x0ffa2, content: 0x0151 Flash mem addr: 0x0ffa4, content: 0x0152 Flash mem addr: 0x0ffa6, content: 0x0153 Flash mem addr: 0x0ffa8, content: 0x0154 Flash mem addr: 0x0ffaa, content: 0x0155 Flash mem addr: 0x0ffac, content: 0x0156 Flash mem addr: 0x0ffae, content: 0x0157 Flash mem addr: 0x0ffb0, content: 0x0158 Flash mem addr: 0x0ffb2, content: 0x0159 Flash mem addr: 0x0ffb4, content: 0x015a Flash mem addr: 0x0ffb6, content: 0x015b Flash mem addr: 0x0ffb8, content: 0x015c Flash mem addr: 0x0ffba, content: 0x015d Flash mem addr: 0x0ffbc, content: 0x015e Flash mem addr: 0x0ffbe, content: 0x015f Flash mem addr: 0x0ffc0, content: 0x0160 Flash mem addr: 0x0ffc2, content: 0x0161 Flash mem addr: 0x0ffc4, content: 0x0162 Flash mem addr: 0x0ffc6, content: 0x0163 Flash mem addr: 0x0ffc8, content: 0x0164 Flash mem addr: 0x0ffca, content: 0x0165 Flash mem addr: 0x0ffcc, content: 0x0166 Flash mem addr: 0x0ffce, content: 0x0167 Flash mem addr: 0x0ffd0, content: 0x0168 Flash mem addr: 0x0ffd2, content: 0x0169 Flash mem addr: 0x0ffd4, content: 0x016a Flash mem addr: 0x0ffd6, content: 0x016b Flash mem addr: 0x0ffd8, content: 0x016c Flash mem addr: 0x0ffda, content: 0x016d Flash mem addr: 0x0ffdc, content: 0x016e Flash mem addr: 0x0ffde, content: 0x016f Flash mem addr: 0x0ffe0, content: 0x0170 Flash mem addr: 0x0ffe2, content: 0x0171 Flash mem addr: 0x0ffe4, content: 0x0172 Flash mem addr: 0x0ffe6, content: 0x0173 Flash mem addr: 0x0ffe8, content: 0x0174 Flash mem addr: 0x0ffea, content: 0x0175 Flash mem addr: 0x0ffec, content: 0x0176 Flash mem addr: 0x0ffee, content: 0x0177 Flash mem addr: 0x0fff0, content: 0x0178 Flash mem addr: 0x0fff2, content: 0x0179 Flash mem addr: 0x0fff4, content: 0x017a Flash mem addr: 0x0fff6, content: 0x017b Flash mem addr: 0x0fff8, content: 0x017c Flash mem addr: 0x0fffa, content: 0x017d Flash mem addr: 0x0fffc, content: 0x017e Flash mem addr: 0x0fffe, content: 0x017f Page 1: Flash mem addr: 0x10000, content: 0x00 Flash mem addr: 0x10001, content: 0x01 Flash mem addr: 0x10002, content: 0x02 Flash mem addr: 0x10003, content: 0x03 Flash mem addr: 0x10004, content: 0x04 Flash mem addr: 0x10005, content: 0x05 Flash mem addr: 0x10006, content: 0x06 Flash mem addr: 0x10007, content: 0x07 Flash mem addr: 0x10008, content: 0x08 Flash mem addr: 0x10009, content: 0x09 Flash mem addr: 0x1000a, content: 0x0a Flash mem addr: 0x1000b, content: 0x0b Flash mem addr: 0x1000c, content: 0x0c Flash mem addr: 0x1000d, content: 0x0d Flash mem addr: 0x1000e, content: 0x0e Flash mem addr: 0x1000f, content: 0x0f Flash mem addr: 0x10010, content: 0x10 Flash mem addr: 0x10011, content: 0x11 Flash mem addr: 0x10012, content: 0x12 Flash mem addr: 0x10013, content: 0x13 Flash mem addr: 0x10014, content: 0x14 Flash mem addr: 0x10015, content: 0x15 Flash mem addr: 0x10016, content: 0x16 Flash mem addr: 0x10017, content: 0x17 Flash mem addr: 0x10018, content: 0x18 Flash mem addr: 0x10019, content: 0x19 Flash mem addr: 0x1001a, content: 0x1a Flash mem addr: 0x1001b, content: 0x1b Flash mem addr: 0x1001c, content: 0x1c Flash mem addr: 0x1001d, content: 0x1d Flash mem addr: 0x1001e, content: 0x1e Flash mem addr: 0x1001f, content: 0x1f Flash mem addr: 0x10020, content: 0x20 Flash mem addr: 0x10021, content: 0x21 Flash mem addr: 0x10022, content: 0x22 Flash mem addr: 0x10023, content: 0x23 Flash mem addr: 0x10024, content: 0x24 Flash mem addr: 0x10025, content: 0x25 Flash mem addr: 0x10026, content: 0x26 Flash mem addr: 0x10027, content: 0x27 Flash mem addr: 0x10028, content: 0x28 Flash mem addr: 0x10029, content: 0x29 Flash mem addr: 0x1002a, content: 0x2a Flash mem addr: 0x1002b, content: 0x2b Flash mem addr: 0x1002c, content: 0x2c Flash mem addr: 0x1002d, content: 0x2d Flash mem addr: 0x1002e, content: 0x2e Flash mem addr: 0x1002f, content: 0x2f Flash mem addr: 0x10030, content: 0x30 Flash mem addr: 0x10031, content: 0x31 Flash mem addr: 0x10032, content: 0x32 Flash mem addr: 0x10033, content: 0x33 Flash mem addr: 0x10034, content: 0x34 Flash mem addr: 0x10035, content: 0x35 Flash mem addr: 0x10036, content: 0x36 Flash mem addr: 0x10037, content: 0x37 Flash mem addr: 0x10038, content: 0x38 Flash mem addr: 0x10039, content: 0x39 Flash mem addr: 0x1003a, content: 0x3a Flash mem addr: 0x1003b, content: 0x3b Flash mem addr: 0x1003c, content: 0x3c Flash mem addr: 0x1003d, content: 0x3d Flash mem addr: 0x1003e, content: 0x3e Flash mem addr: 0x1003f, content: 0x3f Flash mem addr: 0x10040, content: 0x40 Flash mem addr: 0x10041, content: 0x41 Flash mem addr: 0x10042, content: 0x42 Flash mem addr: 0x10043, content: 0x43 Flash mem addr: 0x10044, content: 0x44 Flash mem addr: 0x10045, content: 0x45 Flash mem addr: 0x10046, content: 0x46 Flash mem addr: 0x10047, content: 0x47 Flash mem addr: 0x10048, content: 0x48 Flash mem addr: 0x10049, content: 0x49 Flash mem addr: 0x1004a, content: 0x4a Flash mem addr: 0x1004b, content: 0x4b Flash mem addr: 0x1004c, content: 0x4c Flash mem addr: 0x1004d, content: 0x4d Flash mem addr: 0x1004e, content: 0x4e Flash mem addr: 0x1004f, content: 0x4f Flash mem addr: 0x10050, content: 0x50 Flash mem addr: 0x10051, content: 0x51 Flash mem addr: 0x10052, content: 0x52 Flash mem addr: 0x10053, content: 0x53 Flash mem addr: 0x10054, content: 0x54 Flash mem addr: 0x10055, content: 0x55 Flash mem addr: 0x10056, content: 0x56 Flash mem addr: 0x10057, content: 0x57 Flash mem addr: 0x10058, content: 0x58 Flash mem addr: 0x10059, content: 0x59 Flash mem addr: 0x1005a, content: 0x5a Flash mem addr: 0x1005b, content: 0x5b Flash mem addr: 0x1005c, content: 0x5c Flash mem addr: 0x1005d, content: 0x5d Flash mem addr: 0x1005e, content: 0x5e Flash mem addr: 0x1005f, content: 0x5f Flash mem addr: 0x10060, content: 0x60 Flash mem addr: 0x10061, content: 0x61 Flash mem addr: 0x10062, content: 0x62 Flash mem addr: 0x10063, content: 0x63 Flash mem addr: 0x10064, content: 0x64 Flash mem addr: 0x10065, content: 0x65 Flash mem addr: 0x10066, content: 0x66 Flash mem addr: 0x10067, content: 0x67 Flash mem addr: 0x10068, content: 0x68 Flash mem addr: 0x10069, content: 0x69 Flash mem addr: 0x1006a, content: 0x6a Flash mem addr: 0x1006b, content: 0x6b Flash mem addr: 0x1006c, content: 0x6c Flash mem addr: 0x1006d, content: 0x6d Flash mem addr: 0x1006e, content: 0x6e Flash mem addr: 0x1006f, content: 0x6f Flash mem addr: 0x10070, content: 0x70 Flash mem addr: 0x10071, content: 0x71 Flash mem addr: 0x10072, content: 0x72 Flash mem addr: 0x10073, content: 0x73 Flash mem addr: 0x10074, content: 0x74 Flash mem addr: 0x10075, content: 0x75 Flash mem addr: 0x10076, content: 0x76 Flash mem addr: 0x10077, content: 0x77 Flash mem addr: 0x10078, content: 0x78 Flash mem addr: 0x10079, content: 0x79 Flash mem addr: 0x1007a, content: 0x7a Flash mem addr: 0x1007b, content: 0x7b Flash mem addr: 0x1007c, content: 0x7c Flash mem addr: 0x1007d, content: 0x7d Flash mem addr: 0x1007e, content: 0x7e Flash mem addr: 0x1007f, content: 0x7f Flash mem addr: 0x10080, content: 0x80 Flash mem addr: 0x10081, content: 0x81 Flash mem addr: 0x10082, content: 0x82 Flash mem addr: 0x10083, content: 0x83 Flash mem addr: 0x10084, content: 0x84 Flash mem addr: 0x10085, content: 0x85 Flash mem addr: 0x10086, content: 0x86 Flash mem addr: 0x10087, content: 0x87 Flash mem addr: 0x10088, content: 0x88 Flash mem addr: 0x10089, content: 0x89 Flash mem addr: 0x1008a, content: 0x8a Flash mem addr: 0x1008b, content: 0x8b Flash mem addr: 0x1008c, content: 0x8c Flash mem addr: 0x1008d, content: 0x8d Flash mem addr: 0x1008e, content: 0x8e Flash mem addr: 0x1008f, content: 0x8f Flash mem addr: 0x10090, content: 0x90 Flash mem addr: 0x10091, content: 0x91 Flash mem addr: 0x10092, content: 0x92 Flash mem addr: 0x10093, content: 0x93 Flash mem addr: 0x10094, content: 0x94 Flash mem addr: 0x10095, content: 0x95 Flash mem addr: 0x10096, content: 0x96 Flash mem addr: 0x10097, content: 0x97 Flash mem addr: 0x10098, content: 0x98 Flash mem addr: 0x10099, content: 0x99 Flash mem addr: 0x1009a, content: 0x9a Flash mem addr: 0x1009b, content: 0x9b Flash mem addr: 0x1009c, content: 0x9c Flash mem addr: 0x1009d, content: 0x9d Flash mem addr: 0x1009e, content: 0x9e Flash mem addr: 0x1009f, content: 0x9f Flash mem addr: 0x100a0, content: 0xa0 Flash mem addr: 0x100a1, content: 0xa1 Flash mem addr: 0x100a2, content: 0xa2 Flash mem addr: 0x100a3, content: 0xa3 Flash mem addr: 0x100a4, content: 0xa4 Flash mem addr: 0x100a5, content: 0xa5 Flash mem addr: 0x100a6, content: 0xa6 Flash mem addr: 0x100a7, content: 0xa7 Flash mem addr: 0x100a8, content: 0xa8 Flash mem addr: 0x100a9, content: 0xa9 Flash mem addr: 0x100aa, content: 0xaa Flash mem addr: 0x100ab, content: 0xab Flash mem addr: 0x100ac, content: 0xac Flash mem addr: 0x100ad, content: 0xad Flash mem addr: 0x100ae, content: 0xae Flash mem addr: 0x100af, content: 0xaf Flash mem addr: 0x100b0, content: 0xb0 Flash mem addr: 0x100b1, content: 0xb1 Flash mem addr: 0x100b2, content: 0xb2 Flash mem addr: 0x100b3, content: 0xb3 Flash mem addr: 0x100b4, content: 0xb4 Flash mem addr: 0x100b5, content: 0xb5 Flash mem addr: 0x100b6, content: 0xb6 Flash mem addr: 0x100b7, content: 0xb7 Flash mem addr: 0x100b8, content: 0xb8 Flash mem addr: 0x100b9, content: 0xb9 Flash mem addr: 0x100ba, content: 0xba Flash mem addr: 0x100bb, content: 0xbb Flash mem addr: 0x100bc, content: 0xbc Flash mem addr: 0x100bd, content: 0xbd Flash mem addr: 0x100be, content: 0xbe Flash mem addr: 0x100bf, content: 0xbf Flash mem addr: 0x100c0, content: 0xc0 Flash mem addr: 0x100c1, content: 0xc1 Flash mem addr: 0x100c2, content: 0xc2 Flash mem addr: 0x100c3, content: 0xc3 Flash mem addr: 0x100c4, content: 0xc4 Flash mem addr: 0x100c5, content: 0xc5 Flash mem addr: 0x100c6, content: 0xc6 Flash mem addr: 0x100c7, content: 0xc7 Flash mem addr: 0x100c8, content: 0xc8 Flash mem addr: 0x100c9, content: 0xc9 Flash mem addr: 0x100ca, content: 0xca Flash mem addr: 0x100cb, content: 0xcb Flash mem addr: 0x100cc, content: 0xcc Flash mem addr: 0x100cd, content: 0xcd Flash mem addr: 0x100ce, content: 0xce Flash mem addr: 0x100cf, content: 0xcf Flash mem addr: 0x100d0, content: 0xd0 Flash mem addr: 0x100d1, content: 0xd1 Flash mem addr: 0x100d2, content: 0xd2 Flash mem addr: 0x100d3, content: 0xd3 Flash mem addr: 0x100d4, content: 0xd4 Flash mem addr: 0x100d5, content: 0xd5 Flash mem addr: 0x100d6, content: 0xd6 Flash mem addr: 0x100d7, content: 0xd7 Flash mem addr: 0x100d8, content: 0xd8 Flash mem addr: 0x100d9, content: 0xd9 Flash mem addr: 0x100da, content: 0xda Flash mem addr: 0x100db, content: 0xdb Flash mem addr: 0x100dc, content: 0xdc Flash mem addr: 0x100dd, content: 0xdd Flash mem addr: 0x100de, content: 0xde Flash mem addr: 0x100df, content: 0xdf Flash mem addr: 0x100e0, content: 0xe0 Flash mem addr: 0x100e1, content: 0xe1 Flash mem addr: 0x100e2, content: 0xe2 Flash mem addr: 0x100e3, content: 0xe3 Flash mem addr: 0x100e4, content: 0xe4 Flash mem addr: 0x100e5, content: 0xe5 Flash mem addr: 0x100e6, content: 0xe6 Flash mem addr: 0x100e7, content: 0xe7 Flash mem addr: 0x100e8, content: 0xe8 Flash mem addr: 0x100e9, content: 0xe9 Flash mem addr: 0x100ea, content: 0xea Flash mem addr: 0x100eb, content: 0xeb Flash mem addr: 0x100ec, content: 0xec Flash mem addr: 0x100ed, content: 0xed Flash mem addr: 0x100ee, content: 0xee Flash mem addr: 0x100ef, content: 0xef Flash mem addr: 0x100f0, content: 0xf0 Flash mem addr: 0x100f1, content: 0xf1 Flash mem addr: 0x100f2, content: 0xf2 Flash mem addr: 0x100f3, content: 0xf3 Flash mem addr: 0x100f4, content: 0xf4 Flash mem addr: 0x100f5, content: 0xf5 Flash mem addr: 0x100f6, content: 0xf6 Flash mem addr: 0x100f7, content: 0xf7 Flash mem addr: 0x100f8, content: 0xf8 Flash mem addr: 0x100f9, content: 0xf9 Flash mem addr: 0x100fa, content: 0xfa Flash mem addr: 0x100fb, content: 0xfb Flash mem addr: 0x100fc, content: 0xfc Flash mem addr: 0x100fd, content: 0xfd Flash mem addr: 0x100fe, content: 0xfe Flash mem addr: 0x100ff, content: 0xff ```
technoblogy commented 3 years ago

Mmm.. the lamb sounds good!

MCUdude commented 3 years ago

Mmm.. the lamb sounds good!

It absolutely was!

It turns out I've discovered another bug that seems to have always been there but not yet discovered. When using the sketch I provided in this comment on an ATmega2561 (or an ATmega2560 I'm sure), I can write to any page in the first flash section (0x0000-0xFFFF) and the second section (0x10000-0x1FFFF). However, I try to write to the third section (0x20000-0x2FFFF) or fourth section (0x30000-0x3FFFF) the program crash and goes into a boot loop. I am able to upload new programs with the bootloader, but it has lost its flash writing capabilities, and I'll have to re-burn the bootloader to get to write to flash again. This one probably goes deeper than the other issues I've run into and is probably something @majekw knows more about.

EDIT: It turns out I'm able to erase a page and write to the buffer using optiboot_fill_page, it's only the write command that causes this:

// Function to force a flash page write operation
void flash_end_write(uint32_t base_addr, uint16_t offset_addr)
{
  // Write the buffer to flash if there are any contents in the buffer
  if ((offset_addr & 0xFF) != 0x00)
    optiboot_page_write(base_addr + (offset_addr & (0xFFFF - SPM_PAGESIZE + 1)));
}
technoblogy commented 3 years ago

I thought you might be interested to see what I'm using the Optiboot Flasher for. It's an AVR assembler, written in a combination of C and Lisp, that runs on an ATmega1284P and allows you to write machine-code programs using assembler mnemonics and run them on the processor:

AVR assembler overview

MCUdude commented 3 years ago

@technoblogy I'm not familiar with Lisp, but that is very impressive!

About the issue I was having regarding the 256kiB chips, it turned out that EIND has to be used:

void do_spm_cli(optiboot_addr_t address, uint8_t command, uint16_t data)
{
  uint8_t sreg_save;

  sreg_save = SREG;    // Save old SREG value
  asm volatile("cli"); // Disable interrupts
  #ifdef RAMPZ
    RAMPZ = (address >> 16) & 0xff;  // Address bits 23-16 goes to RAMPZ
    #ifdef EIND
      uint8_t eind = EIND;
      EIND = FLASHEND / 0x20000;
    #endif
    do_spm((address & 0xffff), command, data); // do_spm accepts only lower 16 bits of address
    #ifdef EIND
      EIND = eind;
    #endif
  #else
    do_spm(address, command, data);  // 16-bit address - no problems to pass directly
  #endif
  SREG = sreg_save; // Restore SREG
}

@jandrassy I believe this might be relevant for your copy_flash_pages function as well.

technoblogy commented 3 years ago

Glad you solved it!