Closed wilberforce closed 8 years ago
Further to above, if the line is changed:
if (e) jsError(e);
to
if (e) jsError("error");
so it complies, then the following error occurs:
root@esp8266-VirtualBox:~/Espruino# ./build.sh
CC src/jslex.o
CC src/jsvar.o
CC src/jsvariterator.o
CC src/jsutils.o
CC src/jsnative.o
CC src/jsparse.o
CC src/jspin.o
CC src/jsinteractive.o
CC src/jsdevices.o
CC src/jstimer.o
CC src/jsspi.o
Generating JS wrappers
WRAPPERSOURCES = src/jswrap_array.c src/jswrap_arraybuffer.c src/jswrap_date.c src/jswrap_error.c src/jswrap_espruino.c src/jswrap_flash.c src/jswrap_functions.c src/jswrap_interactive.c src/jswrap_io.c src/jswrap_json.c src/jswrap_modules.c src/jswrap_pin.c src/jswrap_number.c src/jswrap_object.c src/jswrap_onewire.c src/jswrap_pipe.c src/jswrap_process.c src/jswrap_serial.c src/jswrap_spi_i2c.c src/jswrap_stream.c src/jswrap_string.c src/jswrap_waveform.c libs/math/jswrap_math.c libs/network/jswrap_net.c libs/network/http/jswrap_http.c libs/network/js/jswrap_jsnetwork.c libs/network/esp8266/jswrap_esp8266_network.c targets/esp8266/jswrap_esp8266.c libs/network/telnet/jswrap_telnet.c libs/crypto/jswrap_crypto.c
DEFINES = -DGIT_COMMIT=e42ca309cdd9e55cea592bf28ff08cfbfa5c95fb -DBUILDNUMBER="262" -DUSE_DEBUGGER -DUSE_TAB_COMPLETE -DUSE_HEATSHRINK -DUSE_MATH -DUSE_NET -DUSE_ESP8266 -DUSE_TELNET -DUSE_OPTIMIZE_PRINTF -D__ETS__ -DICACHE_FLASH -DXTENSA -DUSE_US_TIMER -DEMBEDDED -DESP8266
CC libs/crypto/jswrap_crypto.o
CC targets/esp8266/spi.o
CC targets/esp8266/user_main.o
CC targets/esp8266/log.o
CC targets/esp8266/jshardware.o
CC targets/esp8266/i2c_master.o
CC targets/esp8266/esp8266_board_utils.o
CC gen/jswrapper.o
LD espruino_esp8266_partial.o
LD espruino_esp8266_user1.elf
LD espruino_esp8266_user2.elf
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user2.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user2.elf] Error 1
make: *** Waiting for unfinished jobs....
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
/home/esp8266/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: address 0x3fffd488 of espruino_esp8266_user1.elf section `.bss' is not within region `dram0_0_seg'
collect2: error: ld returned 1 exit status
make: *** [espruino_esp8266_user1.e
```lf] Error 1
My understanding is that the linked code is too large?
So does something need to be cut back to make it fit?
I'm adding crypto so that I have access to the SHA1 function for use in web sockets.
Ok, it looks like in the error case, the issue is that ESP8266 has to store strings in flash unless you access them in a special way - there's a hack for jsError
because its strings are almost always constant. Try: if (e) jsError("%s", e);
And yes... the other issue you have is probably also related to that. On ESP8266, unless you add special hacks, all constant data is put in RAM. The best solution for now would be to make sure you add "ifndef":"ESP8266"
to the JSON for AES and SHA256/etc functions - so they will not get included and RAM will not get used up for their constant data.
Thanks. The jserror %s hack now compiles.
I tried added the ifndef around the other functions, however still ran out of space.
You could try also looking at libs/crypto/mbedtls/config.h
and removing mention of AES/etc from there.
Hi, I encountered the same issue and sent a pr with a fix (#810).
@wilberforce did you find out how to make it fit?
@urish Sorry, no.
I tried what Gordon suggested with "ifndef":"ESP8266" but it was not enough. I think we need to exclude functions as I thing we are running out of rom space.
@tve any inputs on this one?
I'm considering writing a JavaScript module, based on the sha1.js code that @jumjum has published in the forums.
It could be a standalone require("crypto") to use on boards where sha1 is not in firmware.
This way a separate firmware would not be require and it's only about 1k.
In my use case for web sockets, the module is used during negotiation so could be unloaded once the socket is established.
@urish
If you save this in a projects folder in your web ide, the ws module find this module and uses it:
https://raw.githubusercontent.com/wilberforce/EspruinoDocs/master/modules/crypto.min.js
I'll post to the Forum with an example: http://forum.espruino.com/comments/12817146/
thanks @wilberforce !
In my case, I would like to achieve TLS connections, but I am sure that crypto.js can be useful for other people :)
@urish TLS will need a lot more functions, so I'm not sure if there will be enough space?
No, I think you're out of luck there. In it's default (standards-compliant) mode, mbedtls needs at least 16kB RAM, which is going to mean very little left for Espruino.
If you want TLS, get a Pico and attach an ESP8266 to it. The Pico has enough RAM to handle it.
@wilberforce @gfwilliams thanks for the feedback! I know that the Espressif SDK does have TLS support (I use it with NodeMCU). Do you know whether there was any work done on using their TLS support with Espruino?
No, there wasn't any work done.
IMO it's worth seeing if mbedtls can be tweaked to fit, because then the same code will be used for all Espruino platforms.
If not we're going to end up with a bunch of code that's just left to rot, and that won't get the bugfixes and improvements that the main mbedtls-based code gets.
Just had a quick look at this, and it seems that mbedtls will compile just fine, but as-is there's not enough memory.
You can modify libs/crypto/mbedtls/config.h
to remove features and can then remove bits of libs/crypto/jswrap_crypto.c
that used the other features, and I think it'll then compile.
@gfwilliams You have mentioned the hashlib module will be depreciated in favor of the mdedtls, is there any easy way to remove this from being complied - as that would free up some memory.
I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram? https://github.com/espruino/Espruino/issues/697
I was also looking at the optimisations that @tve had done moving strings to flash, and wondering how much scope there was to free up more ram? Try make topstrings (read the relevant makefile section first)
@tve thanks.
I tired to build using the top strings, but using a recent git pull with the standard Makefile, I can no longer build...
Sorry can't paste here, the error is here:
https://gitter.im/espruino/Espruino?at=56de471268ddef776469772f
It appears that the user1 elf is bigger than 480k and won't fit in the iram1_0_seg _section ?
Seems like it is possible to fit mbedtls into the ESP8266, as it has been done in esp-open-rtos
@tve I have, ./build-topstrings.sh:
root@esp8266-VirtualBox:/home/esp8266/Espruino# cat build-topstrings.sh
#! /bin/bash
export DISABLE_LTO=1
export ESP8266_BOARD=1
export FLASH_4MB=1
export PATH=/home/esp8266/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
export COMPORT=/dev/ttyUSB0
make $*
make clean
./build-topstrings.sh
./build-topstrings.sh topstrings
root@esp8266-VirtualBox:/home/esp8266/Espruino# ./build-topstrings.sh topstrings
Top 20 from ./topstrings:
To get details: xtensa-lx106-elf-objdump -j .rodata.str1.4 -s src/FILENAME.o
root@esp8266-VirtualBox:/home/esp8266/Espruino# ls -l topstrings
-rw-r--r-- 1 root root 0 Mar 13 15:23 topstrings
The file size is 0 - what am I doing wrong?
@wilberforce use xtensa-lx106-elf-objdump -h filename.o to check the used named
in my case it is .rodata.str1.1 and not .rodata.str1.4, so change the Makefile
xtensa-lx106-elf-objdump -h ./src/jswrap_pin.o
./src/jswrap_pin.o: file format elf32-xtensa-le
Sections:
Idx Name Size VMA LMA File off Algn
0 .literal 00000000 00000000 00000000 00000034 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .text 000003ce 00000000 00000000 00000034 2**2
CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
2 .data 00000000 00000000 00000000 00000402 2**0
CONTENTS, ALLOC, LOAD, DATA
3 .bss 00000000 00000000 00000000 00000402 2**0
ALLOC
4 .rodata.str1.1 00000028 00000000 00000000 00000402 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
5 .irom.literal 0000004d 00000000 00000000 0000042c 2**2
CONTENTS, ALLOC, LOAD, DATA
6 .comment 00000022 00000000 00000000 00000479 2**0
CONTENTS, READONLY
7 .xtensa.info 00000038 00000000 00000000 0000049b 2**0
CONTENTS, READONLY
8 .xt.lit 00000048 00000000 00000000 000004d3 2**0
CONTENTS, RELOC, READONLY
9 .xt.prop 000001c8 00000000 00000000 0000051b 2**0
CONTENTS, RELOC, READONLY
Great! Thanks. here is the output:
Top 20 from ./topstrings: 7250 ./espruino_esp8266_partial.o 765 ./src/jsinteractive.o 542 ./libs/network/socketserver.o 512 ./gen/jswrapper.o 438 ./libs/network/esp8266/jswrap_esp8266_network.o 387 ./libs/network/esp8266/ota.o 295 ./src/jslex.o 245 ./src/jsvar.o 225 ./src/jsparse.o 216 ./libs/network/telnet/jswrap_telnet.o 214 ./src/jswrap_process.o 207 ./targets/esp8266/esp8266_board_utils.o 207 ./src/jswrap_json.o 196 ./src/jswrap_io.o 196 ./libs/network/socketerrors.o 176 ./libs/network/esp8266/network_esp8266.o 151 ./targets/esp8266/user_main.o 146 ./src/jswrap_object.o 137 ./src/jswrap_espruino.o 132 ./src/jswrap_pipe.o
In jsinteractive - there is the load banner, and quite a few calls to jsiConsolePrint which I don't think is using the FLASH_STR #define, so quite a few of these strings could go into flash memory
@wilberforce @gfwilliams gfwilliams I have spent some time on it and after reducing the number of js variables from 1400 to 1000 (inside boards/ESP8266_BOARD.py
), it seems to happily compile.
However, make complains about:
** user1.bin uses 529828 bytes of 491520 available
Two questions:
@urish When did you last do a pull, as a recent update of the mathlib reduces the size of the limage:
http://forum.espruino.com/comments/12848811/
This might be small enough.
However, I think you will run out of available RAM as the TLS stuff uses quite large buffers - bigger than the ESP8266 will have available (16K or so from memory). You might be able to compile with a smaller buffer size, perhaps 1K - and see if your https:// server will still connect to this.
One way of testing this might be to use the linux install, and try the smaller packet size with this first.
Is it https:// requests you want to do? You could use a proxy service, or use a pico and esp8266-01, as you won't have the memory limitation then
Yeah, you may run out of memory fot HTTPS (at least without tweaking the config.h to use non-standard (lower than normal) buffer sizes) - however I got the impression you might have just wanted Crypto for the SHA1 and other features - in which case I'd imagine you're fine.
Strange about the high usage from jsinteractive.c - if jsiConsolePrintf could be tweaked with a define the same way the errors have, you might be able to save quite bit of RAM.
Just to add, are you building with RELEASE=1
? That will reduce firmware size a whole load.
As I understand it, you could use more flash, but it would require some linker tweaks and effectively stops you being able to do firmware updates over the air.
@gfwilliams building with RELEASE=1
indeed. Any pointers for for linker tweaks?
@wilberforce building with -lmirom
, is this the new math lib?
For those who need crypto - I managed to compile a firmware just with crypto (without SSL), and the image size seems to fit (haven't tested it yet on the real device, though)
In my case - I want SSL indeed (for HTTPS and secure MQTT), and my use case is presenting the ESP8266 as an IoT capable device on tech talks I am giving. Currently, I use NodeMCU, but most of my audience is not familiar with Lua, so being able to use Espruino can be a game changer in my case.
I'm not sure exactly. @tve might have some ideas, or I think it's just a matter of googling. You need to spread the firmware over user1 and user2 areas.
Having said that, you could add the SAVE_ON_FLASH define in the Makefile, and remove the debugger and command completion defines if they were in there. It might save you a bit of space.
Also you can edit libs/crypto/libmbedtls/config.h
(iirc) and remove any types of crypto you think you can do without.
@gfwilliams thanks! SAVE_ON_FLASH
, disabling telnet and disabling MBEDTLS_SSL_CLI_C
, MBEDTLS_SSL_SRV_C
, MBEDTLS_ECP_C
in config.h got it down to 487380 bytes. Let's now see if this actually runs on the device...
@urish please paste the code to show how you disabled....
@gfwilliams @wilberforce good news - crypto seems to work without a problem
>crypto.SHA256("hello world")
=new ArrayBuffer([185, 77, 39, 185, 147, ... 172, 226, 239, 205, 233])
>
about TLS - I am getting this error:
> http.get('https://www.google.com',function(r){console.log(r);})
Connecting with TLS...
ERROR: Unable to create socket
I guess this is related to the SSL buffer size? Any ideas how to reduce them?
@wilberforce here is the diff:
--- a/Makefile
+++ b/Makefile
@@ -498,14 +498,15 @@ USE_NET=1
else ifdef ESP8266_BOARD
EMBEDDED=1
USE_NET=1
-USE_TELNET=1
+SAVE_ON_FLASH=1
+#USE_TELNET=1
#USE_GRAPHICS=1
BOARD=ESP8266_BOARD
# Enable link-time optimisations (inlining across files), use -Os 'cause else we end up with
# too large a firmware (-Os is -O2 without optimizations that increase code size)
ifndef DISABLE_LTO
OPTIMIZEFLAGS+=-Os -std=gnu11 -fgnu89-inline -flto -fno-fat-lto-objects -Wl,--allow-multiple-definition
-#OPTIMIZEFLAGS+=-DLINK_TIME_OPTIMISATION # this actually slows things down!
+OPTIMIZEFLAGS+=-DLINK_TIME_OPTIMISATION # this actually slows things down!
else
# DISABLE_LTO is necessary in order to analyze static string sizes (see: topstring makefile target)
OPTIMIZEFLAGS+=-Os -std=gnu11 -fgnu89-inline -Wl,--allow-multiple-definition
diff --git a/boards/ESP8266_BOARD.py b/boards/ESP8266_BOARD.py
index 493b36a..b19a406 100644
--- a/boards/ESP8266_BOARD.py
+++ b/boards/ESP8266_BOARD.py
@@ -19,7 +19,7 @@ info = {
'espruino_page_link' : 'EspruinoESP8266',
'default_console' : "EV_SERIAL1",
'default_console_baudrate' : "115200",
- 'variables' : 1400,
+ 'variables' : 1000,
'binary_name' : 'espruino_%v_esp8266',
'build' : {
'defines' : [
diff --git a/libs/crypto/mbedtls/config.h b/libs/crypto/mbedtls/config.h
index 1571980..f694ac0 100644
--- a/libs/crypto/mbedtls/config.h
+++ b/libs/crypto/mbedtls/config.h
@@ -53,7 +53,7 @@
#define MBEDTLS_BIGNUM_C
#define MBEDTLS_CIPHER_C
#define MBEDTLS_CTR_DRBG_C
-#define MBEDTLS_ECP_C
+//#define MBEDTLS_ECP_C
#define MBEDTLS_ENTROPY_C
#define MBEDTLS_MD_C
#define MBEDTLS_MD5_C
@@ -65,8 +65,8 @@
#define MBEDTLS_SHA1_C
#define MBEDTLS_SHA256_C
#define MBEDTLS_SHA512_C
-#define MBEDTLS_SSL_CLI_C
-#define MBEDTLS_SSL_SRV_C
+//#define MBEDTLS_SSL_CLI_C
+//#define MBEDTLS_SSL_SRV_C
#define MBEDTLS_SSL_TLS_C
#define MBEDTLS_X509_CRT_PARSE_C
#define MBEDTLS_X509_USE_C
@@ -119,6 +119,8 @@
#define MBEDTLS_PLATFORM_FREE_MACRO(X) jsvFree(X)
+#define MBEDTLS_NO_PLATFORM_ENTROPY
+
#include "mbedtls/check_config.h"
#endif /* MBEDTLS_CONFIG_H */
Do a process.memory(), and there is a also require("ESP8266").getstatus() that shows heap size.
I'll see if I can find the buffer size - it's in the mBed code.
In config.h there is this comment...
You night be able to make it even smaller... Still looking for the buffer size
Here we go:
>require("ESP8266").getState()
={
"sdkVersion": "1.5.0",
"cpuFrequency": 160, "freeHeap": 5456, "maxCon": 10,
"flashMap": "512KB:256/256",
"flashKB": 512,
"flashChip": "0xe0 0x4016"
}
>process.memory()
={ "free": 930, "usage": 70, "total": 1000, "history": 39 }
>
I guess with an out of memory , we aren't going to see it until it has happened!
https://tls.mbed.org/kb/how-to/reduce-mbedtls-memory-and-storage-footprint
Reduce SSL frame buffer
By default, mbed TLS uses a 16k frame buffer to hold data for incoming and outgoing frames. This is what the TLS standard requires. If you control both sides of a connection (Server and Client) you can reduce the maximum frame size to reduce the buffer's needed to store the data. The size of this frame is determined by MBEDTLS_SSL_MAX_CONTENT_LEN. You can safely reduce this to a more appropriate size (like 2k bytes) if:
both sides support the max_fragment_length SSL extension (allowing reduction to < 1k bytes for the buffers. you control both sides of the connection or know the maximum size that will ever be sent in a single SSL/TLS frame.
There are some examples at the bottom of the page, I'm not sure what the minimum setup you need for a https client.
They link to an example:
#define MBEDTLS_SSL_MAX_CONTENT_LEN 1024
I tried reducing MBEDTLS_SSL_MAX_CONTENT_LEN
to 2048, and also reducing the number of js vars to 300, so right now I have about 13kb of free heap, yet the socket creation still fails :-(
This looks like the minimum config you would need: https://github.com/ARMmbed/mbedtls/blob/development/configs/config-mini-tls1_1.h
ERROR: Unable to create socket
To be honest, this sounds like it could be something really basic. Can you request a non-HTTPS page at all?
I guess it could also be something odd about the way networking is implemented on ESP8266. I added the HTTPS stuff in network.c, but it assumes that everything works as it does for Linux/AT/WIZnet and CC3000. I could be the ESP8266 does something strange with the connect function.
This forum page might be useful too http://forum.espruino.com/conversations/277780/
I think minimum is v1.2
@gfwilliams http works, same call with https fails.
@wilberforce thanks for the link! I also tried now tls.connect()
but it fails with the same error:
>require("tls").connect("https://www.google.com:443", function(res){})
ERROR: Unable to create socket
At some point, I also got this message: ERROR: Not enough memory to allocate SSL socket
,
then I reset the chip, and it no longer complains about the memory, just saying again that it is unable to create the socket.
Some nice progress - I got past the "Unable to create socket" error, and now it successfully creates the socket, though, the SSL handshake fails with a strange error code:
>tls.connect({host:'www.google.com',port:443}), esp.getState().freeHeap
Connecting with TLS...
Performing the SSL/TLS handshake...
=2224
=undefined
ERROR: Failed! mbedtls_ssl_handshake returned -0x2
I honestly have no idea what -0x2 means, I tried to look for it in various places - to no avail.
Where is the 2224 coming from?
That's the free heap, note the esp.getState().freeHeap
at the end of the line
Occasionally it returns -0x7200 (which is MBEDTLS_ERR_SSL_INVALID_RECORD
), but most of the times, simply -2
int mbedtls_ssl_handshake_step( mbedtls_ssl_context *ssl )
{
int ret = MBEDTLS_ERR_SSL_FEATURE_UNAVAILABLE;
if( ssl == NULL || ssl->conf == NULL )
return( MBEDTLS_ERR_SSL_BAD_INPUT_DATA );
#if defined(MBEDTLS_SSL_CLI_C)
if( ssl->conf->endpoint == MBEDTLS_SSL_IS_CLIENT )
ret = mbedtls_ssl_handshake_client_step( ssl );
#endif
#if defined(MBEDTLS_SSL_SRV_C)
if( ssl->conf->endpoint == MBEDTLS_SSL_IS_SERVER )
ret = mbedtls_ssl_handshake_server_step( ssl );
#endif
return( ret );
}
I've Not found -0x2 - could be in mbedtls_ssl_handshake_client_step
.
I take it you are using the search in the repository? https://github.com/espruino/Espruino/search?utf8=%E2%9C%93&q=-0x02&type=Code
Why don't you try another host - rather than google?
Have you tried the simple case?
require("http").get("https://www.google.com", function(res) { res.on('data', function(data) { console.log(data); }); });
@wilberforce tried a dozen, include twitter, github - same result
interestingly, some hosts that I run (nginx), return a different error code, -0x7780 (meaning MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE
)
I think that if I could enable debug logging that could help, though, debugging seems to kill the free ram :-(
Adding the the Makefile:
The issue appears to be the flash_str #define:
I tried changing:
#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x
to
#define FLASH_STR(name, x) static char * name __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x
libs/network/esp8266/jswrap_esp8266_network.c:151:3: error: initializer element is not constant wr0, wr1, wr2, wr3, wr4, wr5, wr6, wr7, wr8, wr9, __wr10, ^ libs/network/esp8266/jswrap_esp8266_network.c:151:3: error: (near initialization for 'wifiReasons[0]')
However then it fails here:
What is the best way to solve this?