Illegal instruction with v1.2.8 on WNDR3700 (OpenWrt 18.06.0)

[alexforencich]

Not much else to say. Seems to work on WNDR4300v1, fails with illegal instruction on WNDR3700v2. It appears both are using the same mips_24kc package.

[mwarning]

I don't think I can debug this without hardware. What image did you use (compiled it yourself or downloaded from openwrt.org)? And how did you install Zerotier?

[alexforencich]

I downloaded the latest build of 18.06.0 from OpenWrt and installed zerotier through the web interface, so I presume it came from the OpenWrt package archive for 18.06.0. If there is anything I can do to help debug this, I can certainly run stuff on the router in question.

[mwarning]

I think the problem are stack protection flags used for building ZT. Some flags would need to be tested. But I do not have the time right now.

[zyclonite]

see the same behavior on WNDR3700v2, when i upgrade to zt 1.2.12 i get a seq fault... interesting is... it does not fail immediately, most of the time its works some minutes, creates the interface, joins the network a.s.o. but suddenly crashes

i tried to build everything from scratch as well... no luck yet

@mwarning what stack protection flags are you referring to?

[mwarning]

ZeroTier defines a few: https://github.com/zerotier/ZeroTierOne/blob/master/make-linux.mk#L66 I had to change it a bit to run on my test platform (and thought this would be sufficient..) https://github.com/openwrt/packages/tree/master/net/zerotier/patches

[zyclonite]

built (against 18.06.0) without stack-protection flag works well on a tl-wdr4300-v1

@alexforencich please test this build on your WNDR3700v2 (i cannot reach mine from work) https://router.wsn.at/dist/packages/mips_24kc/packages/zerotier_1.2.12-2_mips_24kc.ipk

[mwarning]

Nice, if this works for @alexforencich as well, then please make a pull request or let me know what you have changed.

[zyclonite]

my tests

• linksys-wrt1900ac (arm_cortex-a9_vfpv3) - working
• tl-wr1043nd-v1 (mips_24kc) - working
• tl-wdr4300-v1 (mips_24kc) - working
• wndr3700v2 (mips_24kc) - seq fault

removing the -fstack-protector flag did not help

[alexforencich]

Tried it, it seems to be segfaulting

[mwarning]

@zyclonite ZT adds -fstack-protector also to CXXFLAGS in its Makefile and various flags depending on the platform. I suspect the problem to be there.

[zyclonite]

i'll do some more tests/tries but i am off for around a week now

[mwarning]

ok, thank you very much for the testing so far!

[zyclonite]

i did a debug build for the zerotier package, maybe someone can test it (my router seems to be broken, power supply and wan port, so i can not test any longer)

https://router.wsn.at/dist/packages/mips_24kc/packages/zerotier_1.2.12-2_mips_24kc-debug.ipk

[mwarning]

@zyclonite can you get me a openwrt-ramips-mt7620-wt3020-8M-squashfs-sysupgrade.bin? I do not have any other platform available for testing right now.

[zyclonite]

@mwarning here you are (including debug build of zt) https://router.wsn.at/dist/ramips/mt7620/openwrt-ramips-mt7620-wt3020-8M-squashfs-sysupgrade.bin

[mwarning]

@zyclonite ZT/image works for me.

[zyclonite]

i fear its only this wndr3700v2 which does not work...

[zyclonite]

got another wndr370v2 router to test during the weekend, tried the debug version... but it looks like it is always crashing at different situations (at least what the consule output states before the crash)

a bit strange was, the debug version has no symbols... so i could not attach gdb... do i need another flag for the build to include debug symbols?

[mwarning]

The package makefile passes a flag to the Zerotier makefile: https://github.com/mwarning/zerotier-openwrt/blob/master/zerotier/Makefile#L42 Maybe the Makefile do not recognize that flag anymore.

[zyclonite]

i think, that's working because i get debug output but still no debug flags... need to check the zt repo...

[alexforencich]

Same problem under openwrt 18.06.1. Zerotier fails to start with 'illegal instruction'.

[qq472870007]

i have met the same problem on my wndr3800 zerotier version is 1.2.8，I try to update to 1.2.10, faild 'zerotier-one -d' will be killed without any log Neither official or compiled myself And my openwrt is also 18.06.1 and HUAWEI hg255d is faild too

[mwarning]

What about openwrt nightly? https://downloads.openwrt.org/snapshots/targets/

[qq472870007]

https://downloads.openwrt.org/snapshots/targets/ar71xx/generic/openwrt-ar71xx-generic-wndr3800-squashfs-sysupgrade.bin （Tue Oct 9 13:37:29 2018 ） && zerotier 1.2.12 the process is been killed too

[zyclonite]

crashes on some other routers as well (tested with tl-wr1043nd v1)... but it can take to several hours until that happens

[mwarning]

@zyclonite do you see the "invalid instruction" in the system log?

[zyclonite]

i'll try to find this out... problem is, this is a remote router and after zt crashed i need to drive there to check (and on avg it happens every after 6 to 8 hours)

[qq472870007]

it seem the problem is solved on the WNDR3800 the nightly version with zt 1.2.12。 but i meet a new problem with huawei HG255d.

[mwarning]

Please open another ticket for the huawei HG255d if it is still present.