Move to xenial from trusty

[HHHartmann]

OK I played around a bit. The tar file is downloaded only partially by wget to cache, then it cannot be accessed at all. Even on the host system I cannot even access the file properties. All I can do is delete it.

When I download the file (wget) to /tmp I can extract it as usual.

The mount parameters of /opt/nodemcu-firmware are exactly the same on both versions.

So it seems to be related to xenial, I don't see how though.

[marcelstoer]

The tar file is downloaded only partially by wget to cache, then it cannot be accessed at all. Even on the host system I cannot even access the file properties. All I can do is delete it.

That is different on macOS. On the host OS I can unpack it just fine.

[HHHartmann]

Somehow my setup was inconsistent. Now it works. I found an interesting detail in tar though. It can store User permissions name and group in the tar file. When extracting by default it uses the extracting users umask. BUT when extracting as root it tries to restore the permissions from the tar file. In this case -r--r--r-- johny/johny which might lead to the wired permissions. This can be overridden by the tar switch --no-same-permissions So change the Makefile or have docker not run as root or change the archive.

If that's the problem at all

[marcelstoer]

This can be overridden by the tar switch --no-same-permissions

This doesn't help but we're close.

[HHHartmann]

In the other issue it was also mentioned that the tar contains links. Do they work under Windows? Ok I will try this afternoon. We might also check if a persistent docker volume has a suitable Filesystem and then use one. Would need to tweak the makefile then to support another toolchain folder.

[marcelstoer]

@jmattsson this is a weird one, really.

Yet, untaring fails to create new files...BECAUSE the unpacked root folder is created as read-only dr-x------ 2 root root 64 Jul 29 19:43 esp32-linux-x86_64-20181106.0. If I make it writeable explicitly I can untar the next level but not beyond because those folders are also created read-only. AND, this behavior is only observed if tar is run in the mounted volume.

I also tried with tar --no-same-owner --no-same-permissions -xJf toolchain-esp32-linux-x86_64-20181106.0.tar.xz which gives the toolchain folder the same permissions as in your tarball: dr-xr-xr-x 2 root root 64 Nov 6 2018 esp32-linux-x86_64-20181106.0 I haven't figured out yet why your tarball contains read-only directories - and why this is an issue only in mounted Docker volumes.

[jmattsson]

Those weird permissions aren't in the tarball. You can verify that for yourself with a tar tvf toolchain-esp8266-linux-x86_64-20181106.0.tar.xz |grep dr-.

It sounds to me that you have a umask 0222 rather than umask 0022 set? Either in the shell or on the mountpoint itself. Check with umask and mount, respectively?

[HHHartmann]

It's probably also not a problem of docker but of running as root where tar runs without the -no* switches by default.

[marcelstoer]

Sorry, I should have mentioned that I had verified that already. umask is 0022 running as root and the tarball directory permissions are weird:

root@10da821d34d3:/tmp/nodemcu-firmware/cache# tar -tvf toolchain-esp32-linux-x86_64-20181106.0.tar.xz|grep dr-
dr-xr-xr-x johny/johny       0 2018-11-06 02:07 esp32-linux-x86_64-20181106.0/
dr-xr-xr-x johny/johny       0 2018-11-06 02:06 esp32-linux-x86_64-20181106.0/lib/
dr-xr-xr-x johny/johny       0 2018-11-06 02:00 esp32-linux-x86_64-20181106.0/lib/ldscripts/
...
root@10da821d34d3:/tmp/nodemcu-firmware/cache# mount|grep /tmp
osxfs on /tmp/nodemcu-firmware type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

What I also verified is that tar on the host OS also fails for the same reasons while the macOS on-board Archive Utility manages to unpack the tarball.

[jmattsson]

Ah, I looked at the 8266 toolchain. Yes, I can see those odd permissions in the 32 on my end too. Let me see if now would be a good time to build & release a new toolchain...

[jmattsson]

Okay, at least the esp32 toolchain has been updated since I built the last toolchains. I'll kick off new builds and pay extra attention to what goes into the tar file this time.

[jmattsson]

Marcel, could you try this version instead? https://github.com/jmattsson/esp-toolchains/releases/tag/linux-x86_64-20190731.0

[marcelstoer]

Yep, that one works (only tested ESP32 untaring).

[jmattsson]

Maybe easiest to just switch over to that, than you digging further into the hole on OS X? What was the end result of the downloaded-but-unable-to-access-it stuff? That seemed completely unrelated to the weird permissions inside the tar.

[marcelstoer]

Thank you for helping out here! I appreciate that.

Maybe easiest to just switch over to that

Yep, looking forward to you patching https://github.com/nodemcu/nodemcu-firmware/blob/dev-esp32/Makefile#L9

the downloaded-but-unable-to-access-it stuff?

Not sure I understand. Regardless of which way I run tar it unpacks the esp32-linux-x86_64-20181106.0 folder w/o write permissions and that's the end of it.

[jmattsson]

Stop selling yourself short, Marcel - I have full confidence in you PR'ing such a change :D

As for the other comment, I was referring to @HHHartmann's first post above.

[loadedtech]

Marcel, could you try this version instead? https://github.com/jmattsson/esp-toolchains/releases/tag/linux-x86_64-20190731.0

for what it's worth, that esp32 version uncompresses without error on OS X (unlike the earlier version).

[loadedtech]

FYI on OSX the updated updated code (commit https://github.com/nodemcu/nodemcu-firmware/commit/7ad6bc1be779b31fc9a3d8577a6273b1fd7f6f17) runs configure-esp32 nicely out of the box. But the build fails quite a way in to the process (for me):

... CC build/lua/lfunc.o CC build/lua/ldump.o AR build/lua/liblua.a make[2]: Leaving directory '/opt/nodemcu-firmware/build/lua' make[2]: Entering directory '/opt/nodemcu-firmware/build/luac_cross' make[3]: Entering directory '/opt/nodemcu-firmware/build/luac_cross' [ dep] /opt/nodemcu-firmware/components/luac_cross/../base_nodemcu/linit.c In file included from /opt/nodemcu-firmware/components/luac_cross/../lua/lua.h:18:0, from /opt/nodemcu-firmware/components/luac_cross/../base_nodemcu/linit.c:12: /opt/nodemcu-firmware/components/luac_cross/../lua/luaconf.h:305:31: fatal error: readline/readline.h: No such file or directory

[marcelstoer]

Yes, I know but it's unrelated to this PR. Thus, I won't be commenting on that further.

@HHHartmann implemented creating LFS images for ESP8266 a while ago.

At that time there was no LFS for ESP32. That was only added just recently by @jmattsson in https://github.com/nodemcu/nodemcu-firmware/pull/2801. I haven't looked into why this breaks the ESP32 build script here. In the end it's just calling make. IIRC the build of current NodeMCU ESP32 in Docker passed once I manually deleted the luac_cross folder.

[jmattsson]

(looks like a missing libreadline-dev package?)

[marcelstoer]

Didn't remember we had the list of required packages neatly documented at https://nodemcu.readthedocs.io/en/dev-esp32/build/#ubuntu.

[jmattsson]

It's a recent addition, thank Javier for it :)

[loadedtech]

FYI The recent update fixed the readline. Now doing a build on OSX throws the following:

[ dep] /opt/nodemcu-firmware/components/luac_cross/luac.c make[3]: No rule to make target '/usr/lib/gcc/x86_64-linux-gnu/4.8/include/stddef.h', needed by '/opt/nodemcu-firmware/build/luac_cross/uzlib_deflate.d'. Stop. make[3]: Leaving directory '/opt/nodemcu-firmware/build/luac_cross' /opt/nodemcu-firmware/components/luac_cross/component.mk:5: recipe for target 'build' failed make[2]: [build] Error 2 make[2]: Leaving directory '/opt/nodemcu-firmware/build/luac_cross' /opt/nodemcu-firmware/sdk/esp32-esp-idf/make/project.mk:552: recipe for target 'component-luac_cross-build' failed make[1]: [component-luac_cross-build] Error 2 make[1]: Leaving directory '/opt/nodemcu-firmware' Makefile:18: recipe for target 'all' failed make: [all] Error 2