search

SmingHub / Sming

Sming - powerful open source framework simplifying the creation of embedded C++ applications.
https://sming.readthedocs.io
GNU Lesser General Public License v3.0
1.48k stars 346 forks source link

Allow override of SPIFF location #201

Closed piperpilot closed 9 years ago

piperpilot commented 9 years ago

Currently SPIFF location is taken from a calculation based on the _flash_code_end variable that is set in the linker. You can manually adjust the SPIFF location by changing that variable, but it is very clumsy and manual. There should be a way to manually set the SPIFF location in user_config and then override it in the makefile to put it where you want.

In addition for devices with large flash, it would be great if the SPIFF routines used SPI_FLASH_READS instead of memory mapped flash that way it could be placed anywhere on the flash, even above the 1M boundary.

flexiti commented 9 years ago

Btw. size of flash back all the time...maybe it will be useful :) (You can add another type, I have only this two - there are a lot of them)

  int32_t Flash_id;
Flash_id = spi_flash_get_id();
if (Flash_id == 0x1640C8)   //C84016 notice: reverse byte order
    Serial.println("FLASH: GIGADEVICE_GD25Q32  32Mbits = 4MBytes");
else
if (Flash_id == 0x1340C8)  //C84013
    Serial.println("FLASH: GIGADEVICE_GD25Q40  4Mbits = 512KBytes");
else
{
    Serial.print("Flash vendor : ");
    Serial.println((unsigned char)Flash_id,16);     
    Serial.print("Flash type: ");
    Serial.print((unsigned char)(Flash_id>>8),16);
    Serial.println((unsigned char)(Flash_id>>16),16);
    Serial.println("More flash info on https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/master/flashchips.h");
}
STASIUK commented 9 years ago

Flash vendor : EF Flash type: 4016 More flash info on https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/master/flashchips.h

What it means?

flexiti commented 9 years ago

You have flash from WINBOND W25Q32 . http://www.elinux.org/images/f/f5/Winbond-w25q32.pdf

Yes , its 32Mbits memory ( 4 Mbytes)

flexiti commented 9 years ago

/*

STASIUK commented 9 years ago

How to expand the memory file system for this memory?

flexiti commented 9 years ago

I posted You answer at #197

anakod commented 9 years ago

@piperpilot I think you are right, manual position selection will be fine feature for some cases. It also related to air updates store location and filesystem overwriting.

gschmottlach commented 9 years ago

I'm interested in @piperpilot's feature request as well. I'm trying to understand how the calculation below works out . . .

spiffs_config spiffs_get_storage_config()
{
    spiffs_config cfg = {0};
    cfg.phys_addr = ( u32_t )flashmem_get_first_free_block_address();
    if (cfg.phys_addr == 0)
        return cfg;
    cfg.phys_addr += 0x3000;
    cfg.phys_addr &= 0xFFFFC000;  // align to 4 sector.
    cfg.phys_size = INTERNAL_FLASH_SIZE - ( ( u32_t )cfg.phys_addr - INTERNAL_FLASH_START_ADDRESS );
...
}

Why is 0x3000 added to the phys_addr?

How do I translate what's done above to what I see in a typical Sming Makefile for the start of the SPIFFs file system, e.g.

SPIFF_START_OFFSET = $(shell printf '0x%X\n' $$(( ($$($(GET_FILESIZE) $(FW_USER_OUT)) + 16384 + 36864) & (0xFFFFC000) )) )

I understand masking so it's on 4KB (sector-sized) boundaries but where do 16384 and 36864 come from? How are they related to the calculations in the SPIFFs spiffs_get_storage_config()?

Any details you could provide would be helpful. I'm trying to use rboot's "BIG_FLASH" option to map separate 1 MB regions for my two ROMs. I'd like each ROM to have their own SPIFFs file system but I'm not sure how I can adjust these SPIFF_START_OFFSET values. Actually, using the mapped flash read/write routines might work better with Richard's (rboot author) BIG_FLASH feature. See the following for details.

piperpilot commented 9 years ago

Glenn,

I struggled with this calculation as well. I went round and round and thought I had it figured out, Unfortunately I can't find my notes where I did all of the math. What worked for me was this...

In the linker file, I hard coded:

_flash_code_end = ABSOLUTE(0x44000);

This is an arbitrary location that I chose that is bigger than the current ROM. Ideally it should probably be larger than the irom_0_0_seg "len" which is 0x6B000 in my linker file, but really it could be anything.

Next, I changed the makefile to generate the offset as:

SPIFF_START_OFFSET = $(shell printf '0x%X\n' $$(( (0x44000 + 16384) & (0xFFFFC000) )) )

I have no idea how the math works...but this works out. My 2nd spiff is placed at:

SPIFF2_LOCATION = $(shell printf '0x%X\n' $$(( ($(ROM1_LOCATION) - 0x2000) + $(SPIFF_START_OFFSET) )) )

Where ROM1_LOCATION is

ROM1_LOCATION = 0x102000

I think I have an idea how we can modify Sming to allow a fixed location...I'll take a stab at it and try to submit a pull request. I'll also try to pull together a good working example of rboot...I now have rboot updating the spiffs as well during OTA update.

gschmottlach commented 9 years ago

I suspect my setup may be a little different than yours. I'm trying to take advantage of Richard's "BIG_FLASH" feature so I have a single linker file (and thus rom image) that gets stored twice (upper/lower flash slot - each slot 2MB although only 1MB is mapped). Since it's a single image its linked for the 0x40202010 address range (although not necessary flashed there). So although I flash two different SPIFFs roms (in the upper/lower regions of the flash after a given slot), when the application roms are mapped (by default) they're likely looking only at the SPIFF image for rom0 in the lower range of the flash. Either I have to find a way to combine the application/spiffs images into a single bin file that fits within the 1MB mappable region (and that rboot can map as part of it's slot selection logic) or figure out a way for the application to determine it's original location in flash (not the mapped address) and compute where the SPIFFs image should be. Not sure what's the best approach but would be interested in understanding more what you did. Sounds like you're not using the "BIG_FLASH" feature - correct?

Also, how did you teach Richard's OTA methods to download and flash something other than a (bootable) application image? Right now it doesn't seem like it can handle an arbitrary (non-bootable) data image (like the SPIFFs bin). It only likes to put things in defined (image) slots. Perhaps a more flexible OTA mechanism would behave like esptool.py and let you flash arbitrary flash regions regardless of whether the bins are application or SPIFF images. Did you implement something like that?

piperpilot commented 9 years ago

Hey Glenn, I do have this working. I am using BIG_FLASH, but instead of chopping the 4MB into 2 chunks, I am just using 2x1MB chunks on the first half of the flash. I did that just because calculating the offsets, etc was easier at the time, but I can easily switch to the model you are using.

As for updating SPIFFS from OTA, rboot-ota has a mechanism to set an absolute position instead of just using a ROM location, specifically for this purpose. You set the rom_slot to FLASH_BY_ADDR and then you set the rom_addr value to the location where you want to flash. Here is the method from my code that does this below. This is "chained" from the ROM update, so after the ROM update callback is done, before doing the switch and reset, it will call this that then does switch and reset in a callback.

` void SpiffOtaUpdate() {

uint8 slot;
rboot_ota *ota;

// create the update structure
ota = (rboot_ota*)os_zalloc(sizeof(rboot_ota));
memcpy(ota->ip, ota_ip, 4);
ota->port = 80;
ota->callback = (ota_callback)SpiffOtaUpdate_CallBack;
ota->request = (uint8 *)os_zalloc(512);

// select rom slot to flash
slot = rboot_get_current_rom();
if (slot == 0) {
    ota->rom_addr = 0x48000;
} else {
    ota->rom_addr = 0x148000;
}
ota->rom_slot = FLASH_BY_ADDR;

// actual http request
os_sprintf((char*)ota->request,
    "GET /%s HTTP/1.1\r\nHost: yourserver.com\r\n" HTTP_HEADER,
    "spiff_rom.bin",
    IP2STR(ota->ip));

// start the upgrade process
if (rboot_ota_start(ota)) {
    Serial.print("Updating...\r\n");
} else {
    Serial.print("Updating failed!\r\n\r\n");
    os_free(ota->request);
    os_free(ota);
}

}`

I am working on tying all of this together and providing a working example based on the HttpServer_configNetwork example. My primary concern is my own project, but I think this will be a huge benefit. Maybe we'll even get the Sming developers to adopt it as the "standard" OTA since it is more flexible and reliable than the current OTA.

gschmottlach commented 9 years ago

Hi Curtis -

Thanks for sharing this code. I forgot that Richard allows you to flash to a specific address. Certainly makes it feasible now to flash random data segments (like those occupied by SPIFFs). From the ESP8266 forum I can see several people will be interested in an example project that integrates all of these components together. Making the relocation of the SPIFFs mount-point so it's dynamic (perhaps both programmatically at run-time and via Makefile changes) would be a great addition. Right now there seems to be too many hard-coded values . . .. understandable while we're figuring things out but I suspect eventually these values need to be parameterized so they're more flexible and dynamic based on the project needs and configuration.

Again . . . I appreciate you sharing your progress . . . it's been helpful to have others working on this problem too.

AutomationD commented 9 years ago

Here's where I took this math from xlfe/spiffy: STARTADDR = (0x10000 + lengthof(iromtext) + 0x4000) & (0xFFFFC000) (0x10000 was changed in sming)

raburton commented 9 years ago

@piperpilot I think this is now mostly fixed by #244. In branch feature/rboot manual spiffs is fully implemented. In the standard build there is no way to pass it through the makefile, but that could be easily added (just see RBOOT_SPIFFS_0 in the Makefile-rboot.mk). Probably best to default to the current calculation, but allow override from Makefile-user.mk.

piperpilot commented 9 years ago

Yes, I believe this issue is resolved by the rboot changes. Glad to see them ALMOST merged. I'll go ahead and close this as I assume the merge will happen any time now.