espressif / ESP8266_NONOS_SDK

ESP8266 nonOS SDK
Other
925 stars 535 forks source link

Upgrading to nonos SDK 3.10 #212

Open scargill opened 5 years ago

scargill commented 5 years ago

I've not upgraded since NONOS SDK 2.1 due to not enough IRAM in later versions. I use RBOOT by Richard Burton - virtually all my functions are in Flash - my code works perfectly in SDK 2.1 - . I've not kept up with the need to define a partition table. Does the latest version (today - 21/02/2019) free up more space and do the docs now simply explain partition table information?

eriksl commented 5 years ago

I wasn’t aware you were supposed to declare them

That's basic ANSI C, not Linux knowledge :-)

If you're going to use a function in another compilation unit (i.e. it's not declared static and it's also used in another .c file), you NEED to have the declaration of the function available in all files that use them. Also the file that defines the function, so the compiler can cross-check the correctness. You usually do that in an include file so don't have to do it many times.

Other functions (only used in the same compilation unit, and therefore declared as "static") don't need the declaration and the compiler won't bother you with it. Only if you leave the "static" out while you actually did mean such a function. Solution is simple then.

To have more robust and maintainable code, I would suggest to add these to your "CFLAGS": -Wall -Wextra. Then you'll see the compiler will spot a whole lot of potential (or actual) mistakes. That's normal, we're all human, after all. Start by resolving all these warnings until none is left.

and I’ve no idea how to turn warnings on and off, though I prefer the relevant silence of not having them on

That is like wanting to shoot in your own foot.

.. an example of that at the many scripts out there some of which I incorporate into “the script”. There are LOTS of warnings about which I can do nothing, I wish I could turn them off.

That's rubbish and nonsense. In my project I have enabled -Wall and -Wextra and a whopping number of 50 additional explicit warnings and there are no warning in the compilation process. I specified -Werror which means any warning found will abort the compilation.

Now I will give you some advice that you will like more I think. GCC has a nasty habit to automatically inline functions, if they're small or only called once. For most code that is beneficial, it will remove the code for the actuall calling of the function, thereby reducing the code size. BUT in our case it also means that the designation (IRAM/IROM) of the function is silently discarded. So what happens is, that you have an IRAM function that calls a function in IROM (because e.g. it's almost never called). The function gets inlined because GCC sees the benefit and you end up with all of the code from the IROM function in IRAM.

Hint: use -fno-inline and choose function to inline manually, where appropriate.

kriegste commented 5 years ago

I recommend also -Wmissing-prototypes -Wstrict-prototypes

scargill commented 5 years ago

Eriksl - it gets more complicated by the moment, I've never touche d any load script - as far as I know, the linker is handled by the MAKEFILE - so now I know I need to put a partition table and pre-init function in user/user_main.c - here's my MAKEFILE - the LDFLAGS LINE already has most of what you suggest..

LDFLAGS = -nostdlib -Wl,--no-check-sections -u Cache_Read_Enable_New, -u call_user_start -Wl,-static -Wl,-Map,foo.map -Wl,-gc-sections

I'll put the missing bits on on the end as your suggestion....

LDFLAGS = -nostdlib -Wl,--no-check-sections -u Cache_Read_Enable_New, -u call_user_start -Wl,-static -Wl,-Map,foo.map -Wl,-gc-sections -Wl,--size-opt

With SDK 2.1 adding that tiny change - Wl,--size-opt" took me down from 7774 text size to 76f4 - so that was worthwhile in itself..

I'll now try the SDK 3.1 and the partition table and pre-init...

define SDK_RF_CAL_ADDR 0x3FB000

define SDK_PHY_DATA_ADDR 0x3FC000

define SDK_PARAM_ADDR 0x3FD000

define SPI_FLASH_SIZE_MAP FLASH_SIZE_32M_MAP_1024_1024

define SDK_PRIV_PARAM_ADDR 0xFC000

define SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM SYSTEM_PARTITION_CUSTOMER_BEGIN

static const partition_item_t p_table[] = { { SYSTEM_PARTITION_BOOTLOADER, 0x0, 0x2000 }, { SYSTEM_PARTITION_RF_CAL, SDK_RF_CAL_ADDR, 0x1000 }, { SYSTEM_PARTITION_PHY_DATA, SDK_PHY_DATA_ADDR, 0x1000 }, { SYSTEM_PARTITION_SYSTEM_PARAMETER, SDK_PARAM_ADDR, 0x3000 }, { SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM, SDK_PRIV_PARAM_ADDR, 0x1000 }, };

void user_pre_init(void) { if(!system_partition_table_regist(p_table, sizeof(p_table)/sizeof(p_table[0]), SPI_FLASH_SIZE_MAP)) { os_printf("system_partition_table_regist fail\r\n"); while(1); } }

Nope, won't fit... so I'm guessing the "load script" wherever that is, needs work AS WELL. I don't know where that is or what it is called I'm afraid.

eriksl commented 5 years ago

Hi Kriegste – somewhat pointless – the point of upgrading is to fit MORE in, not less. Ta.

Actually @kriegste is right - the point is to get something to compile at all, and THEN fit more in :-)

Have you at any point got your code to compile and load using 3.1 regardless of size? Let's get that happening first and then analyse where where the iram is going.

I would suggest that too. Working from a functioning point to a point what's desirable. Some functions might also be functional in IROM, just for testing. Also when you have a project that completely builds, you can inspect the linker map to find out which files have the functions that take up most IRAM and focus there.

eriksl commented 5 years ago

I recommend also -Wmissing-prototypes -Wstrict-prototypes

As far as I know they're in -Wall or at least in -Wextra (which I'd also recommend).

fvpalha commented 5 years ago

I am trying follow your suggestions, but yet do not work. The project compile, but the firmware (ROM) does not run in ESP8266.

What do you mean by that?

Maybe you're running into the same issue I ran into lately. When I generate an image for a 512 KB flash device, the image won't start. If I change it into an "1 MB 512-512" image, it can be flashed just fine and also runs fine. It seems the whole thong crashes before even anything of my code is called (tested and debugged) so my possibilities end there. It's also not rboot's fault, because it also happens with a "plain" image (no rboot involved).

Thank you very much @eriksl.

I will try to use of the "bigflash" feature from rboot.

eriksl commented 5 years ago

@scargill you will have in your "Makefile" "MAKEFILE" or "Makefile" a line that specifies how the project is linked. There will be an option -T followed by a path to a load script (first lines should be similar to what I posted). It's either in the SDK tree and then you will have to make sure the script used is in the SDK v3 directory (and not maybe some leftover SDK v2 tree) or it's somewhere in your project.

If it's in the SDK (the correct version) it should be ok. If it's in your project then you'll have to change it. As far as I know you can't use the vanilla load scripts from the SDK when you're using rboot, BTW.

The load script tells the linker about the IRAM, IROM, DRAM segments and the sections within them. If it's incorrect, your image and data will end up at the wrong addresses (not so probable) or code will end up in the wrong segment (IRAM instead of IROM) and that's important.

Just search for a file that has the text PHDRS in it, if you found it, it most probably is the right file.

How do you have SDK integration BTW? Are you using esp-open-sdk? Is it in your project itself or outside? How do you change between various SDK versions?

eriksl commented 5 years ago

Also more tips for sizing down on IRAM usage:

Last but not least:

Create a function (in iram) like this:

uint32_t user_iram_memory_is_enabled(void);
iram attr_const uint32_t user_iram_memory_is_enabled(void)
{
    return(0);
}

This will make sure IRAM is not used for the heap. I believe it's the default, but you never know.

eriksl commented 5 years ago

@scargill what functions does your project have that need IRAM (and maybe actually not)? Only interrupt handlers, code that run very tightly timed loops and stuff that handles low-level flash need to be in IRAM. IRAM actually consists of two times 32 Kb and the other 32 Kb is used for caching flash memory. Most of the time the flash ("irom") code will already have been fetched into IRAM and will run exactly as IRAM code.

scargill commented 5 years ago

I wasn’t aware you were supposed to declare them

That's basic ANSI C, not Linux knowledge :-)

If you're going to use a function in another compilation unit (i.e. it's not declared static and it's also used in another .c file), you NEED to have the declaration of the function available in all files that use them. Also the file that defines the function, so the compiler can cross-check the correctness. You usually do that in an include file so don't have to do it many times.

Other functions (only used in the same compilation unit, and therefore declared as "static") don't need the declaration and the compiler won't bother you with it. Only if you leave the "static" out while you actually did mean such a function. Solution is simple then.

To have more robust and maintainable code, I would suggest to add these to your "CFLAGS": -Wall -Wextra. Then you'll see the compiler will spot a whole lot of potential (or actual) mistakes. That's normal, we're all human, after all. Start by resolving all these warnings until none is left.

and I’ve no idea how to turn warnings on and off, though I prefer the relevant silence of not having them on

That is like wanting to shoot in your own foot.

.. an example of that at the many scripts out there some of which I incorporate into “the script”. There are LOTS of warnings about which I can do nothing, I wish I could turn them off.

That's rubbish and nonsense. In my project I have enabled -Wall and -Wextra and a whopping number of 50 additional explicit warnings and there are no warning in the compilation process. I specified -Werror which means any warning found will abort the compilation.

Now I will give you some advice that you will like more I think. GCC has a nasty habit to automatically inline functions, if they're small or only called once. For most code that is beneficial, it will remove the code for the actuall calling of the function, thereby reducing the code size. BUT in our case it also means that the designation (IRAM/IROM) of the function is silently discarded. So what happens is, that you have an IRAM function that calls a function in IROM (because e.g. it's almost never called). The function gets inlined because GCC sees the benefit and you end up with all of the code from the IROM function in IRAM.

-Wall is already in there - I've just added -Wextra - looks like my code isnt so bad after all (I've been writing in C since K&R ws first published) as no extra messages or warnings popped up. Adding -Werror causes no stoppage or warnings. All of this of course is under SDK 2.1 as I can;t seem to move on from that.

@scargill you will have in your "Makefile" "MAKEFILE" or "Makefile" a line that specifies how the project is linked. There will be an option -T followed by a path to a load script (first lines should be similar to what I posted). It's either in the SDK tree and then you will have to make sure the script used is in the SDK v3 directory (and not maybe some leftover SDK v2 tree) or it's somewhere in your project.

If it's in the SDK (the correct version) it should be ok. If it's in your project then you'll have to change it. As far as I know you can't use the vanilla load scripts from the SDK when you're using rboot, BTW.

The load script tells the linker about the IRAM, IROM, DRAM segments and the sections within them. If it's incorrect, your image and data will end up at the wrong addresses (not so probable) or code will end up in the wrong segment (IRAM instead of IROM) and that's important.

Just search for a file that has the text PHDRS in it, if you found it, it most probably is the right file.

How do you have SDK integration BTW? Are you using esp-open-sdk? Is it in your project itself or outside? How do you change between various SDK versions?

I'm not seeing an option T in CFLAGS or LDFLAGS in my project ESP-GO here https://bitbucket.org/scargill/esp-go/src/master/

and yes I'm using the SDK - the subject of this GITHUB area only I'm, still on 2.1 - the point of asking questions here - so I can move on..

scargill commented 5 years ago

@scargill what functions does your project have that need IRAM (and maybe actually not)? Only interrupt handlers, code that run very tightly timed loops and stuff that handles low-level flash need to be in IRAM. IRAM actually consists of two times 32 Kb and the other 32 Kb is used for caching flash memory. Most of the time the flash ("irom") code will already have been fetched into IRAM and will run exactly as IRAM code.

With the exception of very few triny functions - one inline as it is time critical for serial RGB LEDs, everything else has the IFA flag on it.

kriegste commented 5 years ago

Your linker scripts are rom0.ld and rom1.ld. I just tried to use them in my project- .text' will not fit in regioniram1_0_seg'.

I compared the files to the original one here: https://github.com/espressif/ESP8266_NONOS_SDK/blob/master/ld/eagle.app.v6.ld Well, the part below SECTIONS is a bit strange. In your linker files there are some lines missing.

In a first step you should try to replace all that is below SECTIONS in your linker files by the lines from the original linker files.

davydnorris commented 5 years ago

@scargill I found same with your code - these are the ones I am now using with your project. I am working on your code at the moment - have found a lot of warnings, and have picked up a few little things that may or may not have caused randomness in your app.

But the one thing I think will help is the updated roms

rom0-1.zip

davydnorris commented 5 years ago

@scargill - OK after a lot of tidying up of code and removing warnings, and after updating your rboot linker rom0.ld files, I have now got a clean compile of your project, and have the following:

LD build/app_1.out
Memory region         Used Size  Region Size  %age Used
    dport0_0_seg:          0 GB         16 B      0.00%
     dram0_0_seg:       57712 B        80 KB     70.45%
     iram1_0_seg:       27384 B        32 KB     83.57%
     irom0_0_seg:      631284 B       960 KB     64.22%
E2 firmware/.bin

So now you've got an SDK 3.1 build and have 5k iram back. I wasn't going to flash the code and test it but I ended up finishing the job:


----------------------------------

Starting in ROM Slot 0...
Reboot reason: hardware reset
GPIO4 and 5 are outputs
Current web programming pin: 0
GPIO13 is a LED indicator
Firmware version 3.1.0.1
SDK Version: 3.1.0-dev(8a853d3)
DHCP Station mode
Now in STATION mode
Web page control is enabled
Waiting for Access Point 'wififirus'

I tried to upload a zip file but it was too big - what's the best way to get it to you?

davydnorris commented 5 years ago

@eriksl @kriegste - apart from a lot of implicit declarations that can affect compiler optimisations, I am pretty sure it was the old .ld files - there's a whole section in 3.1 that moves the SDK library content into iRAM and this wasn't in the existing files.

scargill commented 5 years ago

@scargill - OK after a lot of tidying up of code and removing warnings, and after updating your rboot linker rom0.ld files, I have now got a clean compile of your project, and have the following:

LD build/app_1.out
Memory region         Used Size  Region Size  %age Used
    dport0_0_seg:          0 GB         16 B      0.00%
     dram0_0_seg:       57712 B        80 KB     70.45%
     iram1_0_seg:       27384 B        32 KB     83.57%
     irom0_0_seg:      631284 B       960 KB     64.22%
E2 firmware/.bin

So now you've got an SDK 3.1 build and have 5k iram back. I wasn't going to flash the code and test it but I ended up finishing the job:


----------------------------------

Starting in ROM Slot 0...
Reboot reason: hardware reset
GPIO4 and 5 are outputs
Current web programming pin: 0
GPIO13 is a LED indicator
Firmware version 3.1.0.1
SDK Version: 3.1.0-dev(8a853d3)
DHCP Station mode
Now in STATION mode
Web page control is enabled
Waiting for Access Point 'wififirus'

I tried to upload a zip file but it was too big - what's the best way to get it to you?

I'm very excited DavyNorris... how to get the ZIP to me... pete@scargill.org should do it... if not, Dropbox or similar? Whatever works for you. Thank you.

scargill commented 5 years ago

@davydnorris - obvioudly not seen your 5am (uk time) work - checked earlier files you put up there rom00-1.zip. Replaced mine - with sdk 2.1 - works. Put in the parttition table and preinit I have - SDK 3.1 compiles for the first time but the code doesn't run (at least not with an ILI display on the ESP8266..... but I've no idea what you did after that first upload of rom00-1.zip - I'll wait until I hear from you now... regards

Pete

kriegste commented 5 years ago

-Wmissing-prototypes -Wstrict-prototypes are not in Wall or Wextra! They have helped me a lot and I highly recommend them.

davydnorris commented 5 years ago

@davydnorris - obvioudly not seen your 5am (uk time) work - checked earlier files you put up there rom00-1.zip. Replaced mine - with sdk 2.1 - works. Put in the parttition table and preinit I have - SDK 3.1 compiles for the first time but the code doesn't run (at least not with an ILI display on the ESP8266..... but I've no idea what you did after that first upload of rom00-1.zip - I'll wait until I hear from you now... regards

Pete

Sent you a link - I had to fiddle with a couple of things in the partition table, but I also had to get the size matched between both the table and the compile.

I don't know if it all works but I got enough to prove I had a running flash.

In the process I fixed

I use Cygwin on windows, and gcc 6.4.0 with the latest esptool. You will need to adjust the environment variables in the Eclipse project to point to your compiler and your SDK but it should work from there. There are still a handful of warnings that you could fix if you wanted pretty code that conforms to the standards better (the MQTT library is a big one there with the function names using caps), but you may choose not to.

scargill commented 5 years ago

Hi

I put in SDK 3.1 and adjusted the com port back to my COM19 – but not sure what else you altered that needs reverting..

10:54:29 Build of configuration Default for project esp-go-3

make -j8 rebuild

Cannot run program "make": Launching failed

Error: Program "make" not found in PATH

PATH=[c:/cygwin64/bin;d:/xtools/bin;C:\Python37\Scripts;C:\Python37]

From: davydnorris [mailto:notifications@github.com] Sent: 07 March 2019 10:46 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] Upgrading to nonos SDK 3.10 (#212)

@davydnorris https://github.com/davydnorris - obvioudly not seen your 5am (uk time) work - checked earlier files you put up there rom00-1.zip. Replaced mine - with sdk 2.1 - works. Put in the parttition table and preinit I have - SDK 3.1 compiles for the first time but the code doesn't run (at least not with an ILI display on the ESP8266..... but I've no idea what you did after that first upload of rom00-1.zip - I'll wait until I hear from you now... regards

Pete

Sent you a link - I had to fiddle with a couple of things in the partition table, but I also had to get the size matched between both the table and the compile.

I don't know if it all works but I got enough to prove I had a running flash.

In the process I fixed

I use Cygwin on windows, and gcc 6.4.0 with the latest esptool. You will need to adjust the environment variables in the Eclipse project to point to your compiler and your SDK but it should work from there. There are still a handful of warnings that you could fix if you wanted pretty code that conforms to the standards better (the MQTT library is a big one there with the function names using caps), but you may choose not to.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/212#issuecomment-470477716 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg9kdjGXL5zfN6zCdpZqXBB0oFb8Kks5vUO3WgaJpZM4bHaSu . https://github.com/notifications/beacon/ABzUg-J-9G0KnBU0dZTLLBA8fRAnAVYZks5vUO3WgaJpZM4bHaSu.gif

scargill commented 5 years ago

Ok, I've now changed the environment variables in the project back to my own - and when trying to comile I'm getting this. So near yet so far. Just spent most of the morning on this..

12:04:53 Build of configuration Default for project esp-go-3 make -j8 rebuild " REBUILDING ALL " RM build FS firmware/webpages.espfs cd html; find | ../mkespfsimage/mkespfsimage.exe > ../firmware/webpages.espfs; cd .. OC build/libmain2.a make: stat: build/driver/bmp280.o: Permission denied RM firmware CC driver/bme280.c /bin/sh: ../mkespfsimage/mkespfsimage.exe: No such file or directory c:\Espressif\xtensa-lx106-elf\bin\xtensa-lx106-elf-objcopy.exe:build/libmain2.a: Permission denied make: [build/libmain2.a] Error 1 make[1]: Entering directory `/d/Users/Peter/Desktop/esp-go-3/rboot' make: Waiting for unfinished jobs.... make[1]: stat: /d/Users/Peter/Desktop/esp-go-3/build: Permission denied mkdir -p /d/Users/Peter/Desktop/esp-go-3/build mkdir -p /d/Users/Peter/Desktop/esp-go-3/firmware CC rboot-stage2a.c LD /d/Users/Peter/Desktop/esp-go-3/build/rboot-stage2a.elf Assembler messages: Fatal error: can't create build/driver/bme280.o: No such file or directory make: *** [build/driver/bme280.o] Error 1 E2 /d/Users/Peter/Desktop/esp-go-3/build/rboot-hex2a.h sed -i -e 's/uint32/uint32_t/' -e 's/uint8/uint8_t/' /d/Users/Peter/Desktop/esp-go-3/build/rboot-hex2a.h CC rboot.c LD /d/Users/Peter/Desktop/esp-go-3/build/rboot.elf E2 /d/Users/Peter/Desktop/esp-go-3/firmware/rboot.bin make[1]: Leaving directory `/d/Users/Peter/Desktop/esp-go-3/rboot'

12:04:54 Build Finished (took 1s.551ms)

eriksl commented 5 years ago

-Wall is already in there - I've just added -Wextra - looks like my code isnt so bad after all (I've been writing in C since K&R ws first published) as no extra messages or warnings popped up. Adding -Werror causes no stoppage or warnings. All of this of course is under SDK 2.1 as I can;t seem to move on from that.

If you learned to program in K&R C, be aware of some nasty traps you can fall into when using ANSI C! One well-known is the promotion of types to some standard types which is done in K&R by default and in ANSI C only when required. If you do it wrong, you won't get a warning but it will crash when calling certain functions.

-Werror will make every warning into an error.

Anyway, as much as I like to talk about this for days, it's is all off-topic.

I'm not seeing an option T in CFLAGS or LDFLAGS in my project ESP-GO here https://bitbucket.org/scargill/esp-go/src/master/

I found your linker scripts within a minute. There are here: https://bitbucket.org/scargill/esp-go/src/4aae40a3d330e5a34b322abc5599e5d0ecc0f36a/rom0.ld?at=master&fileviewer=file-view-default for rboot slot 0 and another one, called rom1 for rboot slot 1.

eriksl commented 5 years ago

@scargill what functions does your project have that need IRAM (and maybe actually not)? Only interrupt handlers, code that run very tightly timed loops and stuff that handles low-level flash need to be in IRAM. IRAM actually consists of two times 32 Kb and the other 32 Kb is used for caching flash memory. Most of the time the flash ("irom") code will already have been fetched into IRAM and will run exactly as IRAM code.

With the exception of very few triny functions - one inline as it is time critical for serial RGB LEDs, everything else has the IFA flag on it.

Then I can't understand why you'd fill up your IRAM. There are imho two possibe causes:

eriksl commented 5 years ago

Oh BTW, I'd like to suggest to implement led pixel control using either one of the two UARTs or the I2S (not I2C...) module, then the timing factor is no longer relevant, because handled by the hardware. Several people have done it (including myself) using the UART. Using I2S I am sure it's possible, but I haven't seen an implementation yet.

eriksl commented 5 years ago

@eriksl @kriegste - apart from a lot of implicit declarations that can affect compiler optimisations, I am pretty sure it was the old .ld files - there's a whole section in 3.1 that moves the SDK library content into iRAM and this wasn't in the existing files.

I have been saying that all the time. In this issue and also in earlier issues that Scargill opened.

The reason is actually not really hard to understand. The SDK v3 libraries are compiled with -fdata-sections and -ffunction-sections. That will make a section for each function and blob of data, instead of putting them all together, for a compilation unit, which is the default behaviour. The default behaviour will place all of the code and the data from a compilation unit in the image, whenever one function or data blob from a compilation unit is used in the image, because they're all in section. When you split up the sections and use one section per function, only the functions you actually use, end up in the image. This may not make sense for file in your project, because functions that you don't use, you should simply delete from the project. But for libraries (SDK...), this makes a huge difference.

The one drawback from this approach is that the section names no longer match those defined in the load script. Unrecognised sections will end up in IRAM. So @scargill had all of the SDK's code end up in IRAM, and that's really too big for it, even if you're not using any of it yourself.

So you need to have an alternative load script that defines sections using wildcards, then the sections are recognised and only IRAM sections will end up in IRAM.

@davydnorris great that you actually fixed it for Pete. Myself I rather have a TS understand the issue and fix it themselves, I think there is more learning potential that way (and less work for me ;-))

eriksl commented 5 years ago

-Wmissing-prototypes -Wstrict-prototypes are not in Wall or Wextra! They have helped me a lot and I highly recommend them.

Weird, they should be in imho! What an unpleasant surprise!

kriegste commented 5 years ago

BTW, adding

*libc.a:(.literal .text .literal.* .text.*)

to my linker file helped me save even more space (I use some of their functions).

Edit: Better not put libgcc in there as is leads to instability.

eriksl commented 5 years ago

This is a bit dangerous though. The functions from libgcc are supposed to be in IRAM because they can be called from functions that are in IRAM. In certain conditions it may cause a crash.

AFAIK all of libc is already compiled to IROM. I am not sure, because I am not using any of it. There is a handful of libc functions I need and I have made simple versions of them in my own code (which is sent to IROM). Often versions in glibc are huge because they need to support stuff like locale, 100 older version of the libc ABI etc.

I'd suggest removing the -lc from the command line, see what functions are missing and if it's worth to do the same.

kriegste commented 5 years ago

That is a good idea! I can see that some libc functions are used in (apart from my code) lwip. But that seems to be in irom anyway. I have no time-critical stuff in my code.

But for libgcc, you are right: libphy uses some of these functions (modsi and so on). And libphy is wifi, so, yes, that should be time-critical. I am going to monitor this and pay close attention.

davydnorris commented 5 years ago

Ok, I've now changed the environment variables in the project back to my own - and when trying to comile I'm getting this. So near yet so far. Just spent most of the morning on this..

12:04:53 Build of configuration Default for project esp-go-3 make -j8 rebuild " REBUILDING ALL " RM build FS firmware/webpages.espfs cd html; find | ../mkespfsimage/mkespfsimage.exe > ../firmware/webpages.espfs; cd .. OC build/libmain2.a make: stat: build/driver/bmp280.o: Permission denied RM firmware CC driver/bme280.c /bin/sh: ../mkespfsimage/mkespfsimage.exe: No such file or directory c:\Espressif\xtensa-lx106-elf\bin\xtensa-lx106-elf-objcopy.exe:build/libmain2.a: Permission denied make: [build/libmain2.a] Error 1 make[1]: Entering directory `/d/Users/Peter/Desktop/esp-go-3/rboot' make: Waiting for unfinished jobs.... make[1]: stat: /d/Users/Peter/Desktop/esp-go-3/build: Permission denied mkdir -p /d/Users/Peter/Desktop/esp-go-3/build mkdir -p /d/Users/Peter/Desktop/esp-go-3/firmware CC rboot-stage2a.c LD /d/Users/Peter/Desktop/esp-go-3/build/rboot-stage2a.elf Assembler messages: Fatal error: can't create build/driver/bme280.o: No such file or directory make: *** [build/driver/bme280.o] Error 1 E2 /d/Users/Peter/Desktop/esp-go-3/build/rboot-hex2a.h sed -i -e 's/uint32/uint32_t/' -e 's/uint8/uint8_t/' /d/Users/Peter/Desktop/esp-go-3/build/rboot-hex2a.h CC rboot.c LD /d/Users/Peter/Desktop/esp-go-3/build/rboot.elf E2 /d/Users/Peter/Desktop/esp-go-3/firmware/rboot.bin make[1]: Leaving directory `/d/Users/Peter/Desktop/esp-go-3/rboot'

12:04:54 Build Finished (took 1s.551ms)

There are a couple of problems showing up there. One is because I deleted the mkespfsimage.exe tool so the archive would be smaller. You will need to remake or copy from your own repo.

Others look like permissions problems with read only files. This may be because you're trying to build this on your Desktop, or maybe because your make isn't handling concurrent build jobs. I am assuming you unzipped the archive and you're not kicking off a build inside it.

Go into the Project Properties and turn off the concurrent builds (they work wonderfully for me but you didn't have them enabled and they may not work for you) - this will build everything in order and will let you see the errors in sequence. Once you work them out, you can try turning it back on to speed things up.

One suggestion offered with respect Pete - it would be so worth your time to find a resource to learn about the Eclipse CDT and how all the properties work, and also a tutorial on gcc and GNU make. Writing the code is only half the task - building it in an accurate and efficient way is just as important. If you've been coding since K&R came out then it's definitely time to crank up the toolchain knowledge

davydnorris commented 5 years ago

@kriegste @eriksl @someburner - this is such a useful thread, it belongs somewhere else other than an issue in a Git repo!!

We should move the content to www.esp8266.com - I would love to have it to come back to, and also to have others contribute as well.

someburner commented 5 years ago

A github issue isn't ideal but honestly the formatting on GH is so much more readable and clean. It is unfortunate that info like this is scattered in issues but GH search works pretty well, and there is a lot less noise on GH. I made posts there years ago but after a while stopped visiting since it seemed like majority of the people were just using Arduino and/or ESP32, which seems to still be the case. Maybe it can be revitalized, but with Espressif pushing IDF/RTOS, it just feels like the number of NONOS people is not going to grow and will be drowned out over there. Just my 2c.

And not to derail the conversation, but I see @scargill talking about wanting the "benefits of SDK 3.x" and I am not really convinced there are any until WPA2-Enterprise works with MSCHAP-V2, unless you actually want to use iram as extra dram. Reports of instability with Arduino do not help either, although I don't use Arduino and it could be Arduino-specific.

Also, I don't know if I mentioned it, I probably should have, that I use some hacks from NodeMCU that places everything into irom by default. Then only the iram functions need special attributes.

eriksl commented 5 years ago

My most beneficial advantage of using the v3 SDK is - and that's really ironical in this thread - is that it leaves more IRAM space. I think most of use try to always use the latest version simply to be ready when an update comes that is really required for us.

What does nodemcu do to achieve this? I think they swapped around the sections definitions in the load script? I always planned to do such a thing, but in the meantime I already got used to always adding either "irom" or "iram" (the latter being a no-op) to every function. I agree it's a pain to always type the whole phrase that Espressif uses (and which is misleading too) so I defined my own versions.

I am having no stability issues with v3 at all. And of course I am not using Arduino.

I feel the same way about esp8266.com and we, the more hardcore hobbyists, being scattered around and the thread here for me also feels like coming home. I am occasionaly still visiting esp8266.com but I get disappointed every time. Only in the very beginning, somewhere in 2014, real stuff was discovered and shared. After that, there are two kinds of threads left: "I can't get my esp8266 working" (and they didn't wire it up correctly or are violating all sorts of restrictions in their code AND they did not read any of the previous threads where the answers are already there) and the other being any random question in any random subforum (and that's what irritates me most) and after some questioning, it appears to mean "on Arduino". As like everyone uses Arduino and there's nothing else. I hate Arduino. To get "real" stuff working there is no benefit in using it.

I'd like to have again conversations about low level stuff, like how to minimise IRAM to a minimum (like this) and also what very much has my interest is to share information about "secret" data Espressif doesn't want to give us. Especially hardware registers and what they do. My ultimate goal is to only use code from the SDK that is really necessary (WLAN stuff) and nothing else, because it's rubbish. I hate it that Espressif is now forcing us to IDF/RTOS, I don't want more Espressif code, I want LESS. Why can't we have a sort of ATMega with a WLAN module concept. With a datasheet of 250 pages describing all registers and how to use them, so we wouldn't be dependent on ANY code from the manufacturer. That's my ultimate goal.

END OF RANT :-)

eriksl commented 5 years ago

BTW a nice example of the above is that Espressif remains silent in all languages to my request as what is the size of the stack. Such a simple question. Nobody including Espressif would or could tell me. I think this is rather an important trait to know about. So now I made a "stack painting" mechanism and I have a clue. But this is really something a vendor should mention in their datasheet.

eriksl commented 5 years ago

I have requested the esp8266.com forum admin to create a new subforum, just for advanced developers discussion (i.e. not babble about crashing images due to people not knowing what they're doing, Arduino, RTOS, etc). I would like to meet there, discuss what we're doing and learn from each other. I still have many questions but I also might be able to contribute some things.

someburner commented 5 years ago

Yeah I agree with many of your points, although I will say those involved in the Arduino project have contributed quite a few cool hacks that can be used in other NONOS projects, namely lwip2, the continuation stack, and the undocumented 16KB of memory, some of which is not used by the SDK.

I'd be interested in an "advanced topics" section as well. This could actually encourage discussion by filtering out the off-topic / repeat questions. Would have to be moderated though.

davydnorris commented 5 years ago

I have requested the esp8266.com forum admin to create a new subforum, just for advanced developers discussion (i.e. not babble about crashing images due to people not knowing what they're doing, Arduino, RTOS, etc). I would like to meet there, discuss what we're doing and learn from each other. I still have many questions but I also might be able to contribute some things.

Fantastic - that's a great start!

I also have found 3.x to be very stable, and the secure networking is finally actually working, which I am really happy about. I'm not a hobbyist (well, sometimes my job feels like a hobby), I am building small and very cheap IoT devices using the ESP8266 as one of the core platforms, and I started on the NonOS SDK because it was the best developed at the time. I will eventually move to the FreeRTOS based SDK once both esp8266 and esp32 chips share common code, but for now I need to be as close to the metal as possible.

I wish there had been an advanced forum on esp8266.com earlier - I ended up designing my own boards based on endless trolling of the forums and other search results after finding so many little quirks and problems. And I ended up working out how the I2S bus worked for input based on the only I2S example I could find, which was an output example. From there I managed to do a lot of 24bit 48kHz streaming DSP with stereo mics with the little memory there is and without locking up the CPU and tripping the watchdog. And I ended up modelling my own device driver code loosely on the Linux and IIO approach and have a sensor board that can be swapped in and out and upgraded in a few seconds, but it took a lot of time!

I shudder to think of the amount of time I wasted doing it all myself! So I am happy to give back if I can.

eriksl commented 5 years ago

@someburner Totally agree. It would preferrably be moderated, indeed.

I also hear stories about the Arduino guys having found out and implemented interesting stuff. So that is what would interest me, not the thousands of Arduino users themselves, that's my point.

LWIP as a whole I would like to discuss, very much.

I don't know about a "continuation stack" but when I was trying to find out what's the actual stack size, about a year and a half ago, I found out there is unused memory at 0x3fffe000 stretching to 0x3fffeb2c, which is not 16 kB, but still quite a bit (2860 bytes). If you skip a few bytes there, there is even more unused memory (another ~ 1500 bytes), but some bytes are overwritten in the initial proces, still beyond the user2_init code. I suspect it's some WLAN related SDK code that uses it. So use with care.

I reported all this to Espressif, asking for some information, but never got a reaction. Typical.

I am very curious where, apparently, the Arduino found another unused 16 Kb, I can hardly believe it.

eriksl commented 5 years ago

Maybe we can share our information about the memory layout and make a more complete, updated map. The ones available now are outdated and incomplete. I'd like that :-)

eriksl commented 5 years ago

@davydnorris I am very curious to your experience with I2S, that's one of the features I haven't yet gotten around actively using. Besides the hardware SPI interface, I guess that's the only one left to explore for me.

Let's see what the reaction is from esp8266.com.

I guess when they agree, I'll scan this issue for topic to be discussed.

scargill commented 5 years ago

@davydnorris we could be getting there -- I copied my mkespfsimage folder into the new project - then turned off the concurrent build as I noticed I don't use it on my original project. Almost there.

CC mqtt/ringbuf.c CC mqtt/utils.c CC user/aidan_and_petes.c user/aidan_and_petes.c: In function 'iprintf': user/aidan_and_petes.c:17:3: error: implicit declaration of function 'ets_vsnprintf' [-Werror=implicit-function-declaration] ets_vsnprintf(buf, sizeof(buf), fmt, args); ^ cc1.exe: all warnings being treated as errors make: *** [build/user/aidan_and_petes.o] Error 1

11:12:04 Build Finished (took 27s.828ms)

eriksl commented 5 years ago

Yeah, yet another one of those dreaded not properly declared functions from the SDK (Espressif's fault).

Most of us have an include file that lists all missing prototypes, mine is like this. There is far more, I think, useful stuff there, but let's keep it ontopic for the moment:

// prototypes missing and undocumented ROM functions

typedef struct {
  uint32_t i[2];
  uint32_t buf[4];
  unsigned char in[64];
  unsigned char digest[16];
} MD5_CTX;

typedef struct {
    unsigned int h0, h1, h2, h3, h4;
    unsigned int Nl, Nh;
    unsigned int data[16];
    unsigned int num;
} SHA_CTX;

struct tm
{
  int   tm_sec;
  int   tm_min;
  int   tm_hour;
  int   tm_mday;
  int   tm_mon;
  int   tm_year;
  int   tm_wday;
  int   tm_yday;
  int   tm_isdst;
};

int ets_vsnprintf(char *, size_t, const char *, va_list);
struct tm *sntp_localtime(const time_t *);

int MD5Init(MD5_CTX *context);
int MD5Update(MD5_CTX *context, const void *, unsigned int length);
int MD5Final(unsigned char *hash, MD5_CTX *context);

int SHA1Init(SHA_CTX *context);
int SHA1Update(SHA_CTX *context, const void *, unsigned int length);
int SHA1Final(unsigned char *md, SHA_CTX *context);

enum { SHA_DIGEST_LENGTH = 20 };

// prototypes missing

double pow(double, double);
double fmax(double, double);

// functions missing from SDK libmain (but declared in headers)

/* int isxdigit(int c); */
/* void *memchr(const void *s, int c, size_t n); */

I have found the last two are used by the SDK but missing because I don't link libc, so I need to define them myself.

scargill commented 5 years ago

NOT getting there, That onr error was just the tip of the iceberg. I tried putting in the definitions for ets_vsnprintf and that caused havoc so I took them back out and took away -WError from the compiler flags - which turned the ets_vsn issue back to a simple warning.. .

user/aidan_and_petes.c: In function 'iprintf':
user/aidan_and_petes.c:17:3: warning: implicit declaration of function 'ets_vsnprintf' [-Wimplicit-function-declaration]
   ets_vsnprintf(buf, sizeof(buf), fmt, args);
CC rboot/appcode/rboot-api.c
In file included from rboot/appcode/rboot-api.c:9:0:
c:\espressif\esp8266_sdk_31/include/spi_flash.h:35:2: error: unknown type name 'uint32'
  uint32 deviceId;
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:36:2: error: unknown type name 'uint32'
  uint32 chip_size;    // chip size in byte
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:37:2: error: unknown type name 'uint32'
  uint32 block_size;
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:38:2: error: unknown type name 'uint32'
  uint32  sector_size;
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:39:2: error: unknown type name 'uint32'
  uint32  page_size;
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:40:2: error: unknown type name 'uint32'
  uint32  status_mask;
  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:45:1: error: unknown type name 'uint32'
 uint32 spi_flash_get_id(void);
 ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:46:41: error: unknown type name 'uint16'
 SpiFlashOpResult spi_flash_erase_sector(uint16 sec);
                                         ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:47:34: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_write(uint32 des_addr, uint32 *src_addr, uint32 size);
                                  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:47:51: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_write(uint32 des_addr, uint32 *src_addr, uint32 size);
                                                   ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:47:69: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_write(uint32 des_addr, uint32 *src_addr, uint32 size);
                                                                     ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:48:33: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_read(uint32 src_addr, uint32 *des_addr, uint32 size);
                                 ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:48:50: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_read(uint32 src_addr, uint32 *des_addr, uint32 size);
                                                  ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:48:68: error: unknown type name 'uint32'
 SpiFlashOpResult spi_flash_read(uint32 src_addr, uint32 *des_addr, uint32 size);
                                                                    ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:52:3: error: unknown type name 'uint32'
   uint32 src_addr,
   ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:53:3: error: unknown type name 'uint32'
   uint32 *des_addr,
   ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:54:9: error: unknown type name 'uint32'
         uint32 size);
         ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:56:30: error: unknown type name 'user_spi_flash_read'
 void spi_flash_set_read_func(user_spi_flash_read read);
                              ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:58:1: error: unknown type name 'bool'
 bool spi_flash_erase_protect_enable(void);
 ^
c:\espressif\esp8266_sdk_31/include/spi_flash.h:59:1: error: unknown type name 'bool'
 bool spi_flash_erase_protect_disable(void);
 ^
rboot/appcode/rboot-api.c: In function 'rboot_get_config':
rboot/appcode/rboot-api.c:35:2: warning: implicit declaration of function 'spi_flash_read' [-Wimplicit-function-declaration]
  spi_flash_read(BOOT_CONFIG_SECTOR * SECTOR_SIZE, (uint32_t*)&conf, sizeof(rboot_config));
  ^
rboot/appcode/rboot-api.c: In function 'rboot_set_config':
rboot/appcode/rboot-api.c:57:2: warning: implicit declaration of function 'spi_flash_erase_sector' [-Wimplicit-function-declaration]
  spi_flash_erase_sector(BOOT_CONFIG_SECTOR);
  ^
rboot/appcode/rboot-api.c:58:2: warning: implicit declaration of function 'spi_flash_write' [-Wimplicit-function-declaration]
  spi_flash_write(BOOT_CONFIG_SECTOR * SECTOR_SIZE, (uint32_t*)((void*)buffer), SECTOR_SIZE);
  ^
make: *** [build/rboot/appcode/rboot-api.o] Error 1

11:23:03 Build Finished (took 38s.937ms)

The build has NEVER had problems with uint32, bool or others...

eriksl commented 5 years ago

I'd like to see what "havoc" you got from adding the proper declration of ets_vsnprintf. I am pretty sure the error just disappeared and the compiler now advances and runs into other issues.

The build has NEVER had problems with uint23, bool or others...

Yeah I can image that. I a guess you mean uint32 ;-)

You're running into another one of the dreaded Espressif misguided actions.

In the v1 and v2 SDK they're using user types called uint8, int8, uint32 etc. They're not standard and just typedefs to unsigned char, long etc. There is really no use for doing this. In SDK v3, they switched (finally) to the standardised integer types as found in the header . There types are defined like uint32_t int8_t and they're useful, and standardised.

This means, apparently, the bogus Espressif types are no longer supplied in the SDK include files. Unfortunately, rboot has also adapted the use of these bogus types, so now you have issues compiling the rboot stuff. I had that as well and fixed it, in my own fork of rboot.

See here what I had to change to make it work again: https://github.com/eriksl/rboot/commit/3be164efd4f631abd1980fa1e9099656a43596c4 You might also ask Richard Burton to do it if course, you never know.

There are other errors as well, I think they come from missing includes in either your code or in rboot's code. Simply search all include files (from the SDK) where these symbols are defined/declared and add the include file to the relevant source. This is NOT the fault of the v3 SDK, you were probably already missing these includes, but you where saved by include files including other include files, which one may never rely on.

davydnorris commented 5 years ago

OK yeah that function is the one function still missing from the SDK includes. I added it directly into my SDK include file osapi.h: int ets_vsnprintf(char *str, unsigned int size, const char *format, va_list argptr);

The other errors are because the SDK header spi_flash.h doesn't use standard data types, and doesn't include c_types.h to resolve them. I have changed them all to standard types, but you can shut them up by adding #include <c_types.h> to the top of spi_flash.h

eriksl commented 5 years ago

I went even further and kludged the function prototypes to be actually correct:

// ugly kludge for incorrectly declared spi_flash_* functions */
#undef spi_flash_write
#define spi_flash_read _spi_flash_read
#define spi_flash_write _spi_flash_write
#include <spi_flash.h>
#undef spi_flash_read
#undef spi_flash_write
SpiFlashOpResult spi_flash_read(uint32_t src, void *dst, uint32_t size);
SpiFlashOpResult spi_flash_write(uint32_t dst, const void *src, uint32_t size);

This relies on the spi_flash.h file to be included always only here (or after this kludge).

scargill commented 5 years ago

I did mean uint32 – I’ve updated the comment… I don’t understand rboot at ALL well, I simply use it… I wonder if somewhere I could stick in definitions which make it work….. uint32 should not be too hard…

From: Erik Slagter [mailto:notifications@github.com] Sent: 08 March 2019 11:57 To: espressif/ESP8266_NONOS_SDK ESP8266_NONOS_SDK@noreply.github.com Cc: Peter Scargill pete@scargill.org; Mention mention@noreply.github.com Subject: Re: [espressif/ESP8266_NONOS_SDK] Upgrading to nonos SDK 3.10 (#212)

I'd like to see what "havoc" you got from adding the proper declration of ets_vsnprintf. I am pretty sure the error just disappeared and the compiler now advances and runs into other issues.

The build has NEVER had problems with uint23, bool or others...

Yeah I can image that. I a guess you mean uint32 ;-)

You're running into another one of the dreaded Espressif misguided actions.

In the v1 and v2 SDK they're using user types called uint8, int8, uint32 etc. They're not standard and just typedefs to unsigned char, long etc. There is really no use for doing this. In SDK v3, they switched (finally) to the standardised integer types as found in the header . There types are defined like uint32_t int8_t and they're useful, and standardised.

This means, apparently, the bogus Espressif types are no longer supplied in the SDK include files. Unfortunately, rboot has also adapted the use of these bogus types, so now you have issues compiling the rboot stuff. I had that as well and fixed it, in my own fork of rboot.

See here what I had to change to make it work again: eriksl/rboot@ https://github.com/eriksl/rboot/commit/3be164efd4f631abd1980fa1e9099656a43596c4 3be164e You might also ask Richard Burton to do it if course, you never know.

There are other errors as well, I think they come from missing includes in either your code or in rboot's code. Simply search all include files (from the SDK) where these symbols are defined/declared and add the include file to the relevant source. This is NOT the fault of the v3 SDK, you were probably already missing these includes, but you where saved by include files including other include files, which one may never rely on.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/espressif/ESP8266_NONOS_SDK/issues/212#issuecomment-470904444 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABzUg3zKUOn3IpY22s5nGWrSskn7pm2Uks5vUlAkgaJpZM4bHaSu . https://github.com/notifications/beacon/ABzUg9CepZjPS3488igiI-HEcXYXKjHjks5vUlAkgaJpZM4bHaSu.gif

scargill commented 5 years ago

Well done @davydnorris - I did those two changes... it could be there is only one more.. help..

CC driver/bme280.c
In file included from user/include/aidan_and_petes.h:14:0,
                 from driver/bme280.c:26:
c:\espressif\esp8266_sdk_31/include/osapi.h:32:69: error: unknown type name 'va_list'
 int ets_vsnprintf(char *str, unsigned int size, const char *format, va_list argptr);
                                                                     ^
make: *** [build/driver/bme280.o] Error 1
kriegste commented 5 years ago

va_list needs

include

eriksl commented 5 years ago

I did mean uint32 – I’ve updated the comment… I don’t understand rboot at ALL well, I simply use it… I wonder if somewhere I could stick in definitions which make it work….. uint32 should not be too hard…

If you're not going to change the code in any way (which is not really possible if you're using it as a submodule from the original url), you'll have to add -D stuff in the makefile. -Duint32=uint32_t etc. To the CFLAGS that are use for compiling the RBOOT stuff. It may be the same as your own project or not. I am not completely sure how to force a source to including a file (stdint.h) without altering the source, you will need it.

And please do me favour. Trim the whole history of the conversation before you hit "send".