search

espressif / ESP8266_NONOS_SDK

ESP8266 nonOS SDK
Other
929 stars 535 forks source link

How to use rBoot with v3.0 ? #163

Closed fvpalha closed 6 years ago

fvpalha commented 6 years ago

Hi.

How I can to configure the partition table in FOTA with rBoot (https://github.com/raburton/rboot) ?

I thinked:

if ((SPI_FLASH_SIZE_MAP == 0) || (SPI_FLASH_SIZE_MAP == 1))

error "The flash map is not supported"

elif (SPI_FLASH_SIZE_MAP == 2)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0xfb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0xfc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0xfd000

elif (SPI_FLASH_SIZE_MAP == 3)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x1fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x1fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x1fd000

elif (SPI_FLASH_SIZE_MAP == 4)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x82000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x3fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x3fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x3fd000

elif (SPI_FLASH_SIZE_MAP == 5)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x102000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x1fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x1fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x1fd000

elif (SPI_FLASH_SIZE_MAP == 6)

define SYSTEM_PARTITION_OTA_SIZE 0x6A000

define SYSTEM_PARTITION_OTA_1_ADDR 0x02000

define SYSTEM_PARTITION_OTA_2_ADDR 0x102000

define SYSTEM_PARTITION_RF_CAL_ADDR 0x3fb000

define SYSTEM_PARTITION_PHY_DATA_ADDR 0x3fc000

define SYSTEM_PARTITION_SYSTEM_PARAMETER_ADDR 0x3fd000

else

error "The flash map is not supported"

endif

@raburton, do you have a tip?

Thank you.

scargill commented 6 years ago

I am using BigFlash. Never fully understood why but I am. I use 1 meg sections – the first and second megs – for the two OTA versions. I use the top end of meg 4, don’t use the third meg at all. RBOOT DOES work and I’d rather keep using it if possible.

In recent versions of my software I’ve reliably OTA’s many times.

Pete

From: Erik Slagter [mailto:notifications@github.com] Sent: 12 November 2018 16:41 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

Using RBOOT doesn't add any code to the IRAM section, unless you're using the BIGFLASH option, then it adds about 150 bytes, not worth mentioning.

Rboot hasn't change much over the last few years and that's because it just works. Just a few extra options have been added. For instance it can now fetch from RTC memory what slot needs to be booted, making it possible to implement a fail-safe boot. Very convenient when you're OTA flashing and you can't easily reach the device.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-437948431 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUgyFF94syoWt6U2RoaGTwH4y0rKCSks5uuaSrgaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUg2bRPmWcOdRRy-zTR0QYTvwLO-kdks5uuaSrgaJpZM4WLOuH.gif

eriksl commented 6 years ago

In the RBOOT documentation you can see BIGFLASH is required to have just one image for all slots. If you're not using BIGFLASH, you must create separate images for every slots, with different loader scripts so they're running at different addresses.

It isn't too difficult to crash the ESP8266 with a bug. If your code has a bug that crashes the ESP8266, you won't be able to flash it OTA, the ESP8266 will keep trying to boot and crash again. That's when you're glad you have the fallback scheme where the older slot is automatically booted whenever the newly slot can't complete booting.

scargill commented 6 years ago

As well as making source available I make BIN files available, just one RBOOT file (at 0), my code at 2000h and that’s it, that gets uploaded to either the first meg slot or the second – the same file.

If the OTA fails it a does go back to the previous version. And all of that works quite well.

Regards

Pete

From: Erik Slagter [mailto:notifications@github.com] Sent: 12 November 2018 18:26 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

In the RBOOT documentation you can see BIGFLASH is required to have just one image for all slots. If you're not using BIGFLASH, you must create separate images for every slots, with different loader scripts so they're running at different addresses.

It isn't too difficult to crash the ESP8266 with a bug. If your code has a bug that crashes the ESP8266, you won't be able to flash it OTA, the ESP8266 will keep trying to boot and crash again. That's when you're glad you have the fallback scheme where the older slot is automatically booted whenever the newly slot can't complete booting.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-437982230 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg61GF6QEDRaV_7K54v1GCZ5U2QAZks5uub1JgaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUgwwp-oFXl5_Vg-2SWNXGv2Seil2_ks5uub1JgaJpZM4WLOuH.gif

scargill commented 6 years ago

Though I don’t follow the details it does make sense that they would be linking in smaller chunks of object code rather than entire files?

From: Erik Slagter [mailto:notifications@github.com] Sent: 12 November 2018 17:52 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

This, I quote myself from a few posts back:

A lot of the space is taken up bij the SDK code BUT the exact amount has been varying between releases and currently it's quite not the most, as Espressif has taken of lot of effort reducing the SDK IRAM footprint. One of the efforts to achieve that was to create linkage units for every separate function instead of having a linkage unit per source file. That means that if you only need one function from a source file, the linker won't draw in all the other functions from the same source file. This has made a huge difference. BUT for that to work, you need to change your linker script, Espressif has shown how to do that. If you fail to do that, you will end up with LESS instead of MORE free IRAM space.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-437972002 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg6xsttsxhGe9ccuarhCVaR96N4mPks5uubVagaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUgxbL9Barjl7esLf2-yzdAmvwfwyTks5uubVagaJpZM4WLOuH.gif

scargill commented 6 years ago

Kind of assumes a lot. Do you mean linker code in my Makefile or a separate file somewhere?

And that begs the question, WHERE have Espressif shown how to do this? I don’t think I know what a “linkage unit” is..

Pete

From: Erik Slagter [mailto:notifications@github.com] Sent: 12 November 2018 17:52 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

This, I quote myself from a few posts back:

A lot of the space is taken up bij the SDK code BUT the exact amount has been varying between releases and currently it's quite not the most, as Espressif has taken of lot of effort reducing the SDK IRAM footprint. One of the efforts to achieve that was to create linkage units for every separate function instead of having a linkage unit per source file. That means that if you only need one function from a source file, the linker won't draw in all the other functions from the same source file. This has made a huge difference. BUT for that to work, you need to change your linker script, Espressif has shown how to do that. If you fail to do that, you will end up with LESS instead of MORE free IRAM space.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-437972002 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg6xsttsxhGe9ccuarhCVaR96N4mPks5uubVagaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUgxbL9Barjl7esLf2-yzdAmvwfwyTks5uubVagaJpZM4WLOuH.gif

kriegste commented 6 years ago

rboot talks about making your own ld files: https://github.com/raburton/rboot#linking-user-code based on the original files: https://github.com/espressif/ESP8266_NONOS_SDK/tree/master/ld Have you tried that?

scargill commented 6 years ago

I can see from above that the relevant file to grab is eagle.app.v6.ld - and I kno where that will need to go. Not entirely sure I follow all the changes that need to be made to the file, however.

This is the current eagle.app.6.ld that sits in the rboot directory. Do I tske it you're suggesting taking a clean one from the Espressif Github and make hte relevat mods whatever they are to THAT file instead?

kriegste commented 6 years ago

Apparently only one line needs to be changed. The one beginning with "irom0_0_seg". It's the address of the OTA image. And this is the only difference in all the ld files anyway. You need two copies of the ld file (one for each OTA image, with their respective addresses). Maybe just take these addresses from your old files... should not be too hard.

scargill commented 6 years ago

I tried taking the Espressif eagle.app.v6.ld file and putting that into the ROOT directory, replacing the original - then editing the one line as R Burton suggests. That netted me a response: eagle.rom.addr.v6.ld: no such file or directory. I've no idea why it says that.

kriegste commented 6 years ago

In the last line, there is an include command. Maybe just correct the path and try again. That is, correct the path so that it points to the (newer) eagle.rom.addr.v6.ld in your SDK folder. Or just put eagle.rom.addr.v6.ld into the same folder...

scargill commented 6 years ago

Yes i do because Richard Burton also changed to a local version of that file. As you say also update the ROM file in RBOOT. Fixed that - and now I get...

c:/espressif/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: build/app_0.out section.text' will not fit in region iram1_0_seg'

scargill commented 6 years ago

So near yet so far.

scargill commented 5 years ago

Can someone help. I use Richard Burton's ROOT and almost all of my functions are in Flash. I've not been able to upgrade the OS since NONOS 2.1 due to IRAM not enough space problems. Code works perfectly in SDK 2.1 and earlier - I note today latest updates to SDK v 3.10 - but I've never had to mess with partitions and hence don't know how. My code (using RBOOT) is used with ESP-12 and I use RBOOT for OTA. Works a treat with 2.1 but I'm running short of RSM for more functions and would like to upgrade to latest SDK for future expansion. Will the latest SDK work with RBOOT (at 0) and my code at 2000h... and can anyone advise on pre-init and partition table info needed, I compile on a PC using the unofficial CHERTS development kit and edit in Eclipse.

fvpalha commented 5 years ago

@scargill, I am searching an alternative to RBOOT. At moment I am studing the project esp_nano_httpd_fota. My develop environment is Linux, with esp-open-sdk upgrade the NONOS SDK 2.2.x.

raburton commented 5 years ago

@scargill you don't need to mess with the partition stuff: https://github.com/espressif/ESP8266_NONOS_SDK/issues/159#issuecomment-419338547

scargill commented 5 years ago

Richard, so glad you are about. Others have said you need to understand the partitioning and some are not very understanding of those of us who don’t eat, sleep and dream about Espressif. I’m good with my code but know nothing about the finer workings of Espressif designer’s thinking. RBOOT just works so I don’t mess with it.

SO what steps/changes do I need to go through to move from SDK 2.1 use to current? All attempts up to now have led to being out of space for functions – iRAM won’t fit or similar. Won’t compile. It would make my day to be able to move on. I’m not out of space yet but quite close which stops me getting ambitious – even though I have loads of ram for normal use.

Pete

From: Richard Antony Burton [mailto:notifications@github.com] Sent: 21 February 2019 14:09 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

@scargill https://github.com/scargill you don't need to mess with the partition stuff: #159 (comment) https://github.com/espressif/ESP8266_NONOS_SDK/issues/159#issuecomment-419338547

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-466012408 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUgwuVzSL6WOlBUL0Y_YWT0w7L4kn9ks5vPqiCgaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUg_c2bMtpjK8mJq_qF2mR8sKOrNduks5vPqiCgaJpZM4WLOuH.gif

scargill commented 5 years ago

I finally got RBOOT and my code running on SDK 3.1 - I hsve noticed sometimes while doing the OTA, what I assume are memory errors... E.M.xxxx - where xxxx is a number... and the OTA fails... no harm done and trying again usually gets it. I have no idea why I see these errors - I'm assuming some kind of timeout issue - anyone any idea what they are for and how to stop that happening?

kriegste commented 5 years ago

That is an out of memory (ram) error.

scargill commented 5 years ago

I just checked - I have a heap of 17k before WIFI connects - then 7688 bytes pretty consistently in normal use... don't understand why 5800 for an OTA would be an issue and why only sometimes... ideas anyone? This time I got EM:5864 followed by EM:5832 - and the OTS failed. I rebooted the board, tried again, this time it worked, no problems.... I wish I understood the memory better...is reducing string storage likely to make any difference? I get no problems with the boards otherwise and thanks to the SDK3.1 update I have more IRAM than ever before.

raburton commented 5 years ago

I just checked - I have a heap of 17k before WIFI connects - then 7688 bytes pretty consistently in normal use... don't understand why 5800 for an OTA would be an issue and why only sometimes... ideas anyone?

heap fragmentation

scargill commented 5 years ago

While I understand the general idea, I've no idea if there's anything I can do about that...

eriksl commented 5 years ago

I have been "living" with a free heap size of around 4500 bytes for quite some time, no problem. Only when I got any lower than that, I started having the E:M messages and failures here and there.

I am using rboot, but not the "update" part. I think the combination of the update code and the Espressif netconn code waste quite a bit of memory by copying the payload. This is exactly the thing I have been addressing lately in my own code, by ditching the Espressif netconn code and using LWIP directly. It now does send chunks of 1460 (MSS) bytes at most now, so in theory that's all you'd need to have free in the heap. Then, if that succeeds (more heap remains available) more chunks are sent in parallel, so the TCP window can increase from 0, which will greatly speed up the connection.

eriksl commented 5 years ago

Maybe Richard or you can implement the same using UDP, which will use much less memory.

scargill commented 5 years ago

I had 7k then I removed an unused static array this week and the heap (as reported by the SDK) went up from 7k to 11k and still I am (sometimes) getting these E.M. messages when attempting OTS. When I get them, the OTS fails, ifI don’t get them, the OTA succeeds. I don’t understand why I am getting these given the same starting conditions – i.e. reboot – connect to WIFI and mqtt as normal (automatic) – I don’t see them any other time. Right now I’m attempting the OTA by serial command – could just as easily do it by mqtt. Maybe half the time success, the rest of the time failure. The percentage is no different to before I saved that RAM by scrapping the array. I do think the failure rate is much higher than it was under SDK 2.1 but that’s maybe coincidental.

Pete

From: Erik Slagter notifications@github.com Sent: 01 April 2019 19:36 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] How to use rBoot with v3.0 ? (#163)

I have been "living" with a free heap size of around 4500 bytes for quite some time, no problem. Only when I got any lower than that, I started having the E:M messages and failures here and there.

I am using rboot, but not the "update" part. I think the combination of the update code and the Espressif netconn code waste quite a bit of memory by copying the payload. This is exactly the thing I have been addressing lately in my own code, by ditching the Espressif netconn code and using LWIP directly. It now does send chunks of 1460 (MSS) bytes at most now, so in theory that's all you'd need to have free in the heap. Then, if that succeeds (more heap remains available) more chunks are sent in parallel, so the TCP window can increase from 0, which will greatly speed up the connection.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/163#issuecomment-478694360 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg1_Dkot42iiDQDFkjJg8xbtx_Wx3ks5vclGFgaJpZM4WLOuH . https://github.com/notifications/beacon/ABzUg_TOVGy8tjSP8VhEHYPBlc_V6pZjks5vclGFgaJpZM4WLOuH.gif

kriegste commented 5 years ago

Do a Google search on "user_iram_memory_is_enabled". Maybe this can help you.

scargill commented 5 years ago

That's a new one on me, Kriegste, I found this.. https://github.com/espressif/ESP8266_NONOS_SDK/issues/211 seems to suggest there's an issue with that option.

kriegste commented 5 years ago

Never had a problem with this. In fact, I have to enable this option to be able to use mbedtls. I also have a web server running (http) and OTA as well. Without this option I never have enough memory for everything to work at the same time.

scargill commented 5 years ago

Ok, thanks for that. As must be obvious I'm a little vague on RBOOT.. I'll give this a shot - that's what backups are for. I'll also try to make the rboot buffer static now that I have 4k more than I had 2 days ago thanks to spotting an array that never got used.

scargill commented 5 years ago

Do a Google search on "user_iram_memory_is_enabled". Maybe this can help you.

It din't - the only reference to that function I could find included the function returning CONFIG_ENABLE_IRAM_MEMORY which I don;t seem to have.

scargill commented 5 years ago

I added user_iram_memory_is_enabled() to user_main - and put the 'define CONFIG_ENABLE_IRAM_MEMORY into user_config.h

Everything compiled and flashed but made no difference at all to running - EM: 5832 (sometimes, not always) when OTAing.

Heap 21472 if I send serial request while waiting for WIFI - 11784 after.... as before (and 5k more than originally when I had an used array in there) - same result though.

5 attempts one after another to use OTA (serial command) (possibly more than usual) - EM error - 6th attempt - got it - swapped ROM slot.

There HAS to be a way to narrow this down?

eriksl commented 5 years ago

I don't think this options is going to help Peter as he is short on IRAM already.

Besides that, IRAM has different alignment demands than DRAM, so it can easily get you a misaligned exception (+ reset).

I think there is "something" with the "out of the box" OTA upload code. I can't blame Richard, it looks like it's more like a playground, a base from where you create your own upload code.

Anyway, I did exactly that so I can't help on this.

raburton commented 5 years ago

5 attempts one after another to use OTA (serial command) (possibly more than usual) - EM error - 6th attempt - got it - swapped ROM slot. There HAS to be a way to narrow this down?

Did you try what I suggested here: https://github.com/raburton/rboot/issues/50#issuecomment-478672239 ?

I think there is "something" with the "out of the box" OTA upload code. I can't blame Richard, it looks like it's more like a playground, a base from where you create your own upload code.

Yes. rBoot is a bootloader, it allows you to do OTA but it doesn't do OTA itself. The rboot-sample project includes a simple OTA example, but it's intended as a basis for your own implementation really (not to say that it can't potentially be used as is).

scargill commented 5 years ago

Bear in mind I've been using this same project with SDK 2.1 and earlier for years and most of the time the RBOOT OTA worked perfectly ( quite a few people using the code - ESP-GO. I then moved to SDK 3.1 after some great help in here which got my IRAM back - now I have loads of it... and as for RAM I gained back 4k just this week by commenting out an array that never ended up being used... but it is since moving to SDK 3.1 that I've had these issues with OTA, Richard, you may recall some years ago you helped me get RBOOT to work. I've not touched that folder since.

raburton commented 5 years ago

Sounds like something has changed with SDK 3.1 in terms of how malloc works, or how much it's being called. If you malloc and free a lot, or you create and destroy lots of objects, you have the potential for fragmentation. The problem with your code is that it's made up of loads of bits of other peoples code stuck together and you don't know what most of it is doing with memory, just what your resulting total free heap size is. Eliminating wasted memory to give you a bigger heap and reducing mallocs (reusing buffers and objects instead of freeing and creating again) will reduce the risk of being unable to allocate a large buffer, but not eliminate it. Reducing the size of the malloc you are trying to make here, if possible, will also help. The problem is you don't know what even know what malloc is failing, so like I said in the comment referenced above - find that first and you'll know what the code is actually trying to do when it fails and what you might do about it. I've suggested one place in rBoot where there can be a large malloc (if you pass in a large amount of data to write at once from your OTA code), and if it is there how to reduce it (simply pass in less at a time).

scargill commented 5 years ago

I think you are right about SDK 3.1 - as my version of RBOOT code has not changed or been updated for over a year (at least). I went in to look at the rboot_flash_write code you mentioned and could not make head nor tail of how the buffer size was calculated. The function appears to be only called once.... and LEN at the end is always 4. There wasn't a hope of understanding it well enough to sensibly meddle with it... so I left it. Also worth noting (I've not mentioned it up to now) is occasional failure with DNS lookup for the OTA address - until now I'd assumed this was not relevant - but it is looking like it is all tied together. If OC_MALLOC is used outside of RBOOT I've not noticed and as the bulk of project code is my own in user/petes_code.c I would know if I'd used it - I haven't.

So, I've taken recent changes - and updated my SDK 2.1 code.... compiled, flashed and OTA all on SDK 2.1 - and now it all works - no lookup issues, no memory issues (just as it was before I decide to "upgrade" to SDK 3.1). I'm trying the same board over and over - and up to now no failures with SDK 2.1

As I have no issues with any code outside of OTA/RBOOT on SDK 3.1 - and as I have enough IRAM to be getting on with, with hesitation I've reverted back to 2.1 - if Richard or anyone else IS able to help me narrow this down further then I'll pick up on 3.1 again - right now I feel like I'm banging my head against a brick wall.

eriksl commented 5 years ago

My $0.02: I am not using os_malloc (aka pvPortMalloc etc) in my code, but LWIP does. I see the free heap size shrinking considerately when I send or receive large chunks of data over TCP, but when everything is sent or received, the amount of free heap completely returns to what it was.

Brainstorming:

If ever possible I'd try to use UDP for this, if you want to keep it simple. It needs considerately less memory. Max payload is 1472 bytes IIRC, don't try to make the ESP8266 reassemble fragmented UDP packets, it won't do that (needs much memory). I am using a scheme where the client first, in 1-4 steps, fills a 4096 byte flash sector buffer in the ESP8266 and after that, have it written to flash and continue to the next sector (very much simplified).

kriegste commented 5 years ago

Defining CONFIG_ENABLE_IRAM_MEMORY to 1 should give you at least 16kB more to play with. Is that the case?

raburton commented 5 years ago

I think you are right about SDK 3.1 - as my version of RBOOT code has not changed or been updated for over a year (at least). I went in to look at the rboot_flash_write code you mentioned and could not make head nor tail of how the buffer size was calculated. The function appears to be only called once.... and LEN at the end is always 4.

It's called from two places in the same function in the rboot-sample ota implementation. The length that is passed in is the size of the data received from the network. So if you want to reduce the size of the malloc in this function just pass in less data at a time. Mind you as far as I can tell you still haven't worked out what malloc call is failing, and I've suggested that 3 times now, so I may be wasting my time here. Either way, if you are passing in a lot of data, even if it's not this one that is failing, repeatedly allocating large chunks of memory is going to increase the risk of heap fragmentation.

scargill commented 5 years ago

You are not wasting your time,I'm following as best I can... I'm looking again to see where rboot_write_flash is used. I don;t have any sample files... Are you the sam Erichard that helped me originally, if so you may recall you did the rboot setup for me. I know little about it as I've never had to go in and delve until after SDK 2.1

In APPCODE subfolder, rboot_write_flash is actually defined, then called ONCE ONLY as below in rboot_write_end function (and not in a loop)....

return rboot_write_flash(status, status->extra_bytes, 4);

The LENGTH looks like 4 to me...

In the rboot folder, rboot.c doesn't call it... but NOW I found it in user/rboot_ota.c called twice in the callback function upgrade_recvcb where length is an unsigned short so my next job will be to find out where upgrade_recvcb is used,,,

When you say pass less data as clearly you know this function, is there a figure in the callback function I can play with?

Pete

scargill commented 5 years ago

Defining CONFIG_ENABLE_IRAM_MEMORY to 1 should give you at least 16kB more to play with. Is that the case?

Kriegste - thanks for the feedback - I tried defining that as 1 in include/user_config.h and as far as the heap function is concerned it made no difference at all. Am In looking in the wrong place? I use system_get_free_heap_size())

kriegste commented 5 years ago

Define it to 0, take down the value system_get_free_heap_size() reports. Then define it to 1. Again take down the value. It should be higher. Otherwise it could be there is something wrong with your implementation of the function user_iram_memory_is_enabled()...

scargill commented 5 years ago

I just did an experiment... in SDK 2.1 - adding

define CONFIG_ENABLE_IRAM_MEMORY 1

to include/user_config.h

takes me from a heap of 13,160 to either 13352 or 13,328, a minor improvement but at least it is doing something.

Before the WIFI is ready, the heap is 22,792, when the WIFI is ready the heap drops to 13,352 and shortly thereafter 13,328 where it remains steady.

Not so easy to test in SDK 3.1 as the OTA is not reliable but I had 21k before the WIFI was ready and something like 11k when it was. CONFIG_ENABLE_IRAM_MEMORY did nothing in SDK 3.1.

kriegste commented 5 years ago

Actually, you can implement the function like this:

uint32 user_iram_memory_is_enabled(void) {
    return 1;
}

I suspected the define is used in a header file, but it is not. So adding just this function should be enough.

scargill commented 5 years ago

Define it to 0, take down the value system_get_free_heap_size() reports. Then define it to 1. Again take down the value. It should be higher. Otherwise it could be there is something wrong with your implementation of the function user_iram_memory_is_enabled()...

That function doesn't exist in SDK 2.1 version user/user_main.c - should it or are you referring to the sdk3.1 version?

kriegste commented 5 years ago

2.1 is like stone-age. There is no user_iram_memory_is_enabled.

scargill commented 5 years ago

You're ahead of my I was just replying to your previous comment. OK, shall I put the user_iram_is_enabled function in user/user_main.c and in the SDK2.1, 3.1 or both and should that really be

uint32 ICACHE_FLASH_ATTR user_iram_memory_is_enabled(void) { return 1; }

like all the rest of my functions?

scargill commented 5 years ago

Right, not in 2.1 however adding that define DID give me an extra 180 bytes or thereabouts.... so not entirely a waste.

scargill commented 5 years ago

Herer's a thing - in SDK 3.1 ub user/user_main.c, I'd created the function user_iram_is_enabled and it had no effect - obvious really as it is not being called anywhere - so now I'm creating user_iram_memory_is_enabled which will just return 1 - but then no-one is calling that either. If it already exists somewhere I'll soon know...

scargill commented 5 years ago

Right - my new version of user_iram_memory_is_enabled is not causing any issues - therefore it can't already exist and so can't be in use anywhere - which makes it a tad pointless?

raburton commented 5 years ago

You are not wasting your time,I'm following as best I can... I'm looking again to see where rboot_write_flash is used. I don;t have any sample files...

Your OTA implementation is from the rboot-sample application. As I said before, rBoot is a bootloader, it does not do OTA. There is a sample application (to demonstrate using rBoot which includes a basic OTA implementation) and that's where your OTA code comes from.

Are you the sam Erichard that helped me originally, if so you may recall you did the rboot setup for me. I know little about it as I've never had to go in and delve until after SDK 2.1

And this is part of the problem. You have a lot of components strapped together, sometimes by other people, that you don't understand. If you don't know what's in your app it's hard know where your memory is going or how to debug problems.

In APPCODE subfolder, rboot_write_flash is actually defined, then called ONCE ONLY as below in rboot_write_end function (and not in a loop).... return rboot_write_flash(status, status->extra_bytes, 4); The LENGTH looks like 4 to me...

Yes, these are utility functions that are provided with rBoot for use in users' applications, and which are used by your OTA implementation, so you need to look in your application...

In the rboot folder, rboot.c doesn't call it...

They are not used by rBoot, because rBoot doesn't do OTA.

but NOW I found it in user/rboot_ota.c called twice in the callback function upgrade_recvcb where length is an unsigned short so my next job will be to find out where upgrade_recvcb is used,,,

Bingo, this is your OTA code. Note this is in the user folder, this is part of your app (you didn't write it, obviously), but it's not rBoot it's part of the OTA implementation in your app, which comes from a sample.

When you say pass less data as clearly you know this function, is there a figure in the callback function I can play with?

No, but you can break the data up and pass it in several smaller chunks quite easily.

I must admit that rboot_write_flash isn't written very efficiently. It has to handle data of any size, but only perform writes of multiple of 4 bytes. It does this by copying the data into a new buffer, resulting in a malloc and free, which could be for a large amount of data. It does this even if all the data comes to it in multiples of 4. Even if it didn't, it could be done in such a way as to not need the malloc at all. If SDK 3.1 has a malloc that isn't very good at preventing fragmentation, or the network code is doing something odd that causes slowly increasing sizes of packet each time (which would make these mallocs much more prone to causing fragmentation) this probably would need to be rewritten.