[TW#22830] Would you consider folding in the work done to bring lwIP up to date?

[davydnorris]

David Gauchard aka d-a-v has done some great work implementing a shim layer for the ESP to be able to use the most recent releases of lwIP

https://github.com/d-a-v/esp82xx-nonos-linklayer.git

Given that a lot of the SDK is now open, would you consider merging this effort so that the official SDKs are brought up to date?

[wujiangang]

@davydnorris Thanks, it's a useful info, we will try this.

[eriksl]

+1!

[d-a-v]

To be able to cleanly plug any IP stack on top of the SDK, it would be nice and more efficient that espressif changed the nonos-sdk api to be a real link-layer api without any reference to lwIP from inside the closed-source libraries. That indeed would be a major change.

That's what propose the above shim layer (with a little price). It is not exclusively usable with arduino. There is a quite old PR for esp-open-sdk (still to be polished but it could be there with some help) so virtually anybody could use it as is.

[eriksl]

The SDK code should stay away from layer 3 networking altogether. It should only call some defined hooks in an external layer 3 library, like lwip. But I guess that's what you're saying.

I am going to try the above, on plain NON-OS SDK, not Arduino, so fingers crossed...

[eriksl]

Can't get it to work. The "Makefile.open" make assumptions about the opensdk that don't hold. The opensdk is a structure that from the root has the crosstool-NG, esp-open-lwip, xtensa-lx106-elf (bin...) and the current SDK under it. The makefile is searching for an arch/cc.h somewhere, some include files from SDK for patching and if I have them all fixed up using make variables and symlinks, I got tons of errors compiling, like this:

--- patch-non-local-includes: patching user_interface.h (include "lwip/xxx" -> <lwip/xxx>)
--- patch-non-local-includes: patching sntp.h (include "lwip/xxx" -> <lwip/xxx>)
In file included from /home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/ESP8266_NONOS_SDK/third_party/include/lwip/opt.h:45:0,
                 from /home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/ESP8266_NONOS_SDK/third_party/include/lwip/ip_addr.h:35,
                 from /home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/xtensa-lx106-elf/sysroot/usr/include/user_interface.h:30,
                 from glue/glue.h:41,
                 from glue-esp/lwip-esp.c:35:
/home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/ESP8266_NONOS_SDK/third_party/include/lwipopts.h:793:1: warning: "/*" within comment [-Wcomment]
 /*
 ^
In file included from glue/glue.h:42:0,
                 from glue-esp/lwip-esp.c:35:
glue/esp-missing.h:32:6: error: conflicting types for 'ets_timer_arm_new'
 void ets_timer_arm_new (ETSTimer *a, int b, int c, int isMstimer);
      ^
In file included from glue/gluedebug.h:60:0,
                 from glue/glue.h:33,
                 from glue-esp/lwip-esp.c:35:
/home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/xtensa-lx106-elf/sysroot/usr/include/osapi.h:65:6: note: previous declaration of 'ets_timer_arm_new' was here
 void ets_timer_arm_new(os_timer_t *ptimer, uint32_t time, bool repeat_flag, bool ms_flag);
      ^
glue-esp/lwip-esp.c:63:2: error: #error PBUF_RSV_FOR_WLAN should be defined
 #error PBUF_RSV_FOR_WLAN should be defined
  ^
glue-esp/lwip-esp.c:66:2: error: #error EBUF_LWIP should be defined
 #error EBUF_LWIP should be defined
  ^
In file included from glue-esp/lwip-esp.c:44:0:
glue-esp/lwip-esp.c: In function 'pbuf_wrapper_get':
glue-esp/lwip-esp.c:219:74: error: too many arguments to function 'pvPortMalloc'
   struct pbuf_wrapper* p = (struct pbuf_wrapper*)os_malloc(sizeof(struct pbuf_wrapper) * PBUF_WRAPPER_BLOCK);
                                                                          ^
/home/erik/src/esp/esp8266-universal-io-bridge/lwip/sdk/ESP8266_NONOS_SDK/third_party/include/lwip/mem.h:91:35: note: in definition of macro 'os_malloc'
 #define os_malloc(s) pvPortMalloc(s, "", __LINE__,true)
                                   ^
In file included from glue/glue.h:42:0,
                 from glue-esp/lwip-esp.c:35:
glue/esp-missing.h:13:7: note: declared here
 void* pvPortMalloc (size_t xWantedSize, const char* file, int line) __attribute__((malloc, alloc_size(1)));
       ^
glue-esp/lwip-esp.c: In function 'pbuf_alloc':
glue-esp/lwip-esp.c:641:28: error: too many arguments to function 'pvPortMalloc'
   struct pbuf* p = (struct pbuf*)mem_malloc(alloclen);
                            ^
In file included from glue/glue.h:42:0,
                 from glue-esp/lwip-esp.c:35:
glue/esp-missing.h:13:7: note: declared here
 void* pvPortMalloc (size_t xWantedSize, const char* file, int line) __attribute__((malloc, alloc_size(1)));
       ^
glue-esp/lwip-esp.c:660:28: error: too many arguments to function 'pvPortMalloc'
   struct pbuf* p = (struct pbuf*)mem_malloc(alloclen);
                            ^
In file included from glue/glue.h:42:0,
                 from glue-esp/lwip-esp.c:35:
glue/esp-missing.h:13:7: note: declared here
 void* pvPortMalloc (size_t xWantedSize, const char* file, int line) __attribute__((malloc, alloc_size(1)));
       ^
make[3]: *** [makefiles/Makefile.rules:8: build/glue-esp/lwip-esp.o] Error 1
make[2]: *** [makefiles/Makefile.build-lwip2:47: liblwip2.a] Error 2
make[1]: *** [Makefile.open:12: all] Error 2
make: *** [Makefile:281: lwip] Error 2

[eriksl]

BTW I now see most errors are from updated SDK, where some function's prototypes have been been added recently or protoypes have been changed. Please update to SDK v3.

But these two seem to be severe:

glue-esp/lwip-esp.c:63:2: error: #error PBUF_RSV_FOR_WLAN should be defined
 #error PBUF_RSV_FOR_WLAN should be defined
  ^
glue-esp/lwip-esp.c:66:2: error: #error EBUF_LWIP should be defined
 #error EBUF_LWIP should be defined

[d-a-v]

I am mainly experimenting on the arduino side, and we have not yet gone through the SDK v3 door. The esp-open-sdk port is surely out of date but it shouldn't require a lot of work. Minimal tcp example that I don't have with esp-open-sdk would surely ease that work (I tried to make it work with micropython as a validating PR, which is not the easiest way I reckon). Once the basis is stabilized again, I'll get back to this if no one tries at first.

[eriksl]

Ok, just yell and I'll try. I have absolutely nothing with Arduino. I am also considering switching to lwip (be it the one that Espressif ships or an external one) to begin with, then the glue layer isn't required. But the documention of how to use lwip on ESP8266 is really far too minimal to be useful.

[someburner]

PSA: I have put up a PR here https://github.com/d-a-v/esp82xx-nonos-linklayer/pull/29 that compiles with esp-open-sdk. PR contains instructions for how to build. Note that you currently need to provide your own SNTP implementation- just copy esp-sntp.c into your project somewhere. Tested on linux / Ubuntu 18.04.

[davydnorris]

This is really cool @someburner - I'll have a bash at this after I get my current release out.

Re: sntp, I don't suppose you know whether the ESP code actually updates one of the internal clocks. At the moment I'm just calling sntp_get_timestamp all the time but it would be simpler (and the sntp code above would benefit) if you could just use a clock value instead.

At some stage I want to attack the sntp code above and try to add in some compensation to try and get decent millisecond accuracy - the RTC is not very good in the ESP (really temperature sensitive) but the system clock is great - should be a way to use that to get a proper millis() unless it's already done inside the code somewhere

[eriksl]

BTW I solved the NTP issue in two ways:

• force the SNTP to run every 10 minutes. Actually it should (imho) do that itself, but SNTP is oneshot in ESP8266.
• I have a server that multicasts the complete date/time every minute to all ESP's in the multicast group, so they're always perfectly in sync (and actually don't need SNTP so I turned it off).