How to use rBoot with v3.0 ?

[fvpalha]

Hi.

How I can to configure the partition table in FOTA with rBoot (https://github.com/raburton/rboot) ?

I thinked:

if ((SPI_FLASH_SIZE_MAP == 0) || (SPI_FLASH_SIZE_MAP == 1))

error "The flash map is not supported"

elif (SPI_FLASH_SIZE_MAP == 2)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0xfb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0xfc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0xfd000

elif (SPI_FLASH_SIZE_MAP == 3)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x1fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x1fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x1fd000

elif (SPI_FLASH_SIZE_MAP == 4)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x3fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x3fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x3fd000

elif (SPI_FLASH_SIZE_MAP == 5)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x102000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x1fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x1fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x1fd000

elif (SPI_FLASH_SIZE_MAP == 6)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x102000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x3fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x3fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x3fd000

else

error "The flash map is not supported"

endif

@raburton, do you have a tip?

Thank you.

[scargill]

RABURTON - You're right I didn't write that user/rboot-ota.c file. Unfortunately I don't understand the callback very well - it seems to have a length passed to it but I've no idea how you pass parameters to a callback function or where they are getting passed. Are you suggesting that something could be changed inside that upgrade_recvcb function without knowing the length of what is actually being passed to it?

[raburton]

You don't pass the length to the callback, because you don't call the callback. It's a network receive callback, it's called by the sdk when data is received, and the length of the data is passed to it (along with a pointer to that data). So inside upgrade_recvcb you have the data and know it's length. You can then pass it all at once to the rboot_write_flash function, as the code currently does, or pass it in several smaller chunks.

[scargill]

I know I'm going to look incredibly dense here but I'll go for it anyway. RABURTON - does this look in any way sensible? In the middle of upgrade_recvcb

else { // not the first chunk, process it upgrade->total_len += length; rboot_write_flash(&upgrade->write_status, (uint8*)pusrdata, length); }

That seems to be where the bulk of data is sent to rboot_write_flash. Any idea as to how to split that up?

[raburton]

Just use a loop to call rboot_write_flash several time with chunks of the data. Both the calls in that function have the potential to pass a large amount of data to rboot_write_flash (although the first is only called once) so you'd want to split them both.

[scargill]

I'll have a go. Off to Spain this weekend so if I run out of time it will be mid-week sometime.

Pete

[raburton]

Pete, I've done a rewrite of rboot_write_flash in rboot-api.c which avoids the malloc altogether. Inexplicably you still haven't told us which malloc is actually failing, so it may not be this one. However, it is certainly more efficient and will help reduce heap fragmentation, so wherever your failing malloc actually is, this may still help. Please test this out:

bool ICACHE_FLASH_ATTR rboot_write_flash_internal(rboot_write_status *status, uint8_t *data, uint32_t len) {

    // erase any additional sectors needed by this chunk
    int32_t lastsect = ((status->start_addr + len) - 1) / SECTOR_SIZE;
    while (lastsect > status->last_sector_erased) {
        status->last_sector_erased++;
        spi_flash_erase_sector(status->last_sector_erased);
    }

    // write current chunk
    //os_printf("write addr: 0x%08x, len: 0x%04x\r\n", status->start_addr, len);
    if (spi_flash_write(status->start_addr, (uint32_t *)((void*)data), len) == SPI_FLASH_RESULT_OK) {
        status->start_addr += len;
        return true;
    }

    return false;
}

// function to handle writing to the flash
// call repeatedly with more data (max len per write is the flash sector size (4k))
bool ICACHE_FLASH_ATTR rboot_write_flash(rboot_write_status *status, uint8_t *data, uint16_t len) {

    int32_t extras;

    if (data == NULL || len == 0) {
        return true;
    }

    // if we have any remaining bytes, can we make a block of 4 with the new bytes?
    if (status->extra_count != 0) {
        extras = 4 - status->extra_count;
        if (len >= extras) {
            memcpy(status->extra_bytes + status->extra_count, data, extras);
            data += extras;
            len -= extras;
            if (!rboot_write_flash_internal(status, status->extra_bytes, 4)) return false;
            status->extra_count = 0;
        }
    }

    // check length is a multiple of 4
    // else save any remaining bytes for next go
    extras = len % 4;
    if (extras != 0) {
        len -= extras;
        memcpy(status->extra_bytes + status->extra_count, data + len, extras);
        status->extra_count += extras;
    }

    if (len == 0) {
        return true;
    }

    return rboot_write_flash_internal(status, data, len);
}

[eriksl]

@kriegste to use IRAM for DRAM you need to do BOTH the define and the callback function.

It's not going to solve anything though, as, and I mentioned this before, Peter is already very low on IRAM. The memory must come from "somewhere" it's not free ;)

[scargill]

Not inexplicable, RABURTON - I've not told anyone whichmalloc is failing because I have no idea which malloc is failing. I CAN say that no other part of tthe project fails, just the OTA and only some of the time and only on DSK 3.x - on 2.x it is fine. ERIKSL - I'm no longer that short of IRAM - on SDK 2.1, .text is now 76fa which is still high but better than it was and on SDK 3.1 .text is 6c50 which is a lot better - and it is sdk 3.1 I'm having OTA issues with, not SDK 2.1 which works fine. Granted I could always use more space..... I don't understand how that callback will change anything as things seem to work well without it being there?

Richard, I'm almost out of time as I have to pack for our Spain summer exodus but I'll try your new write_flash routines now... thank you for this effort - much appreciated.

[scargill]

RABURTON

I just tried your new code on SDK 3.1 – with no other changes, commenting out the original rboot_write_flash and replacing with these two functions you've supplied above. I flashed the code onto my board, then made the two ROMS available on the OTA site and attempted OTA - immediate board crash and reboot (reboot reason, fatal exception). The board then continued to work but of course no OTA.

To make sure it wasnt just me being incompetent I commented out the new code, unommented the original, rebuilt and reflashed the board and made roms available on the OTA site and attempted OTA, back to the original E.M.issue which as a reminder doesn't happen every time, attempt 2 worked.

So, something up with the new code. Any chance of a 2nd attempt? There's no chance it was finger trouble on my behalf, I copied/pasted straight from your reply.

Pete

[eriksl]

The complete IRAM area is 32 kbytes. When you enable the iram-as-dram option, you will loose half of it, only 16 kbytes remain available for the "text" section / iram segment. I am pretty sure there will be not many users able to cram all of the SDK "iram" code (mainly that) + their own into 16 kbytes.

I'd rather have Espressif taken 16 kbytes of the IRAM cache area.

[scargill]

Silly question coming up RABURTON, in rboot_write_flash you pass a len parameter which is a 16 bit var, but then you pass that to the internal version which is expecting a 32 bit var, is that a potential for why it is crashing on OTA in this version?

[scargill]

That answers that. I tried changing LEN to 32 bit in each case, then 16 bit in each case. Made no difference, as soon as I initiate OTA the program reboots.

[raburton]

Silly question coming up RABURTON, in rboot_write_flash you pass a len parameter which is a 16 bit var, but then you pass that to the internal version which is expecting a 32 bit var, is that a potential for why it is crashing on OTA in this version?

No the compiler can handle that. Not sure about the crash, obviously not seeing that here. However, all my packets came in at a multiples of 4 bytes in length, so never went through half the new code (designed to handle non-4 byte packets, because we must write to flash in 4 bytes chunks). I'll try and do a bit more testing on that when I get chance...

[scargill]

Thanks RABURTON. I may go quiet until mid-week after tomorrow, travelling from Northern UK to Soutrhern Spain the hard way, no connectivity for part of the journey. I will follow up any success you have with that. Much appreciated.

[eriksl]

I am not quite at ease with writing less-than-complete sectors. If we're lucky, the SDK code does a read-modify-write, but who knows what they really do? In the best case it will wear out your flash sectors very quickly, I'm not sure about the worst case. At least it will be very slow.

So I always write complete 4k sectors.

That's why I implement a scheme where the ota server uploads chunks of 2^n size into a sector buffer and only when the sector is complete (and checksummed with MD5!), it will flash it (and then do a verify read + compare as well).

This scheme does need a dedicated server interface program though. I have been thinking of something using Java or Javascript in a browser, but I think it will be too complex. Any approach using a http post will loose the ability to checksum and verify and it's very hard to implement, due to the multipart handling of http post and the translation of base64 to binary (= will use a lot of memory).