search

lancethepants / tomatoware

Tomatoware is a set of scripts to create a native development environment for tomato firmware supported routers.
Other
71 stars 22 forks source link

Compiling with tomatoware on RPi, "undefined reference to `__fputc_unlocked'" #36

Closed blackfuel closed 7 years ago

blackfuel commented 7 years ago

I'm running tomatoware on a Raspberry Pi, to compile a static version of cryptsetup. When building libgcrypt, it appears that it's not correctly linking the C runtime library. Just want to be sure it's not a problem with uClibc. I'm using a modified version of your cryptsetup.sh build script. It encountered this linker error in the libgcrypt build step. The other packages compiled fine. What's different about libgcrypt?

libtool: link: gcc -g -O2 -fvisibility=hidden -Wall -o dumpsexp dumpsexp-dumpsexp.o
libtool: link: ranlib .libs/libgcrypt.a
libtool: link: rm -fr .libs/libgcrypt.lax
libtool: link: ( cd ".libs" && rm -f "libgcrypt.la" && ln -s "../libgcrypt.la" "libgcrypt.la" )
/bin/bash ../libtool  --tag=CC   --mode=link gcc -I/mmc/include -g -O2 -fvisibility=hidden -Wall   -o mpicalc mpicalc-mpicalc.o libgcrypt.la -L/mmc/lib -lgpg-error
libtool: link: gcc -DSTANDALONE -g -O2 -fvisibility=hidden -Wall -o hmac256 hmac256-hmac256.o
libtool: link: gcc -I/mmc/include -g -O2 -fvisibility=hidden -Wall -o mpicalc mpicalc-mpicalc.o  ./.libs/libgcrypt.a -L/mmc/lib /mmc/lib/libgpg-error.a
mpicalc-mpicalc.o: In function `my_getc':
/mmc/src/cryptsetup/gcrypt/libgcrypt-1.6.6/src/mpicalc.c:282: undefined reference to `__fgetc_unlocked'
mpicalc-mpicalc.o: In function `main':
/mmc/src/cryptsetup/gcrypt/libgcrypt-1.6.6/src/mpicalc.c:553: undefined reference to `__fputc_unlocked'
/mmc/src/cryptsetup/gcrypt/libgcrypt-1.6.6/src/mpicalc.c:545: undefined reference to `__fputc_unlocked'
/mmc/src/cryptsetup/gcrypt/libgcrypt-1.6.6/src/mpicalc.c:407: undefined reference to `__stdout'
collect2: error: ld returned 1 exit status
Makefile:717: recipe for target 'mpicalc' failed
make[2]: *** [mpicalc] Error 1
make[2]: Leaving directory '/home/pi/tomatoware/src/cryptsetup/gcrypt/libgcrypt-1.6.6/src'
Makefile:477: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/pi/tomatoware/src/cryptsetup/gcrypt/libgcrypt-1.6.6'
Makefile:408: recipe for target 'all' failed
make: *** [all] Error 2

To see what's going on, you could try running my build script, that I have forked from your repo: https://github.com/blackfuel/cryptsetup-arm-static/blob/master/cryptsetup.sh

Thank you.

lancethepants commented 7 years ago

Sorry for the late reply, had a massive cold this past week that put me down for the count.

Well this is interesting, never seen anyone running tomatoware on a rpi. I'm assuming you want this for an arm router since all your other repos are firmware related, and not for the rpi.

Well the first thing that I wonder about is tomatoware's interaction with the environment of the rpi. It's possible/probable that in some cases tomatoware is looking at the rpi's library and headers. GCC and a lot of libraries themselves will default look in places like /lib:/usr/lib, and /usr/lib/include, which could definitely foul things up.

My first suggestion would be to run tomatoware in a chroot environment, so it can't be aware of the rest of the pi. I haven't ever really played around with this, but I think it would work. If not in a chroot, then I would make sure on compiling everything you use CPPFLAGS and LDFLAGS to tell gcc where to find the include and libraries files respectively for the tomatoware environment and to prefer them first over anything else.

I just updated my repo with the latest of everything and it compiled fine..... on my router. I tried compiling with your script on my router and it stopped on lvm2, but didn't immediately see why. I can't really replicate your setup. Ok, maybe I could did out my first gen rpi somewhere, but I think it was only single core, so I will stick with my dual core router cpu.

blackfuel commented 7 years ago

Just now I was able to compile with my cryptsetup.sh on the RT-AC68U router (Asuswrt-Merlin 380.66-alpha2). I'm also compiling cryptsetup without --disable-kernel-crypto, so that it's able to mount Veracrypt and Truecrypt formatted devices. For this to work, my script automatically installs a kernel header file into /mmc/include/linux. That change was just checked in now.

My Pi is: Raspberry Pi 2 Model B V1.1 (quad-core ARM Cortex-A7 CPU + 1GB RAM) . It is a hard-float CPU.

I will try again with chroot. My preference is to compile everything on the Pi. Here's the /proc/cpuinfo:

processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 1
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 2
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 3
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : BCM2709
Revision        : a21041
Serial          : 0000000000000000
blackfuel commented 7 years ago

Just updated the PATH variable inside cryptsetup.sh so that all /mmc folders are searched first. It's really that simple!! Now it compiles 100% on the Raspberry Pi 2.

Also, I had to delete the /mmc folder and restore a fresh one because some libraries got installed during my initial attempts at compiling that somehow had RPi dependencies or similar issues. Thank you for this awesome Tomatoware!!

UPDATE: I copied the program files from the RPi2 to the RT-AC68U router. I found that the cryptsetup.static program opens a Veracrypt file container in 2 minutes versus 5 minutes with Entware-ng. Wow!! However, the cryptsetup program had a segmentation fault, I imagine because not all of the packages were compiled for dynamic loading?

lancethepants commented 7 years ago

Hard to say, though if the statically linked binary works, statically linking individual libraries should be ok. Does it work on the pi? When using the dynamically linked binary, does your router have the same tomatoware environment that the pi does? If your binary has dependencies on dynamic libraries you've compiled, you will need those to be present as well on the router. I am a solid advocate of static linked binaries when possible.

blackfuel commented 7 years ago

The cryptsetup program that I compiled with tomatoware does not work correctly on the Pi. No segfault; just nothing shows up in /dev/mapper, after the normal 2 minute wait. However, the cryptsetup.static program runs fine on the Pi. This is surprising since Raspbian is based on Linux 4.4.

Yes, my router has a /mmc tomatoware environment. I also recompiled cryptsetup on the router with the same results, the cryptsetup program returns segfault after a 2 minutes wait, and the cryptsetup.static program runs fine.

There are programs that I need compiled directly into my firmware, and cryptsetup is one. Currently, my modified version of Asuswrt-Merlin has the Entware-ng version of cryptsetup compiled in. It uses dynamic linking and it runs fine. However, I want better performance. Ideally, I would like to compile a version of cryptsetup linked against Asuswrt, however there was a problem linking libuuid in LVM2.

I like statically-linked binaries, however in a firmware it quickly uses up the space.

For comparison, I ran LDD trace utility against both Tomatoware and Entware-ng versions of cryptsetup. Notice that the Entware-ng version has libgcc_s and libm, and tomatoware does not.

ldd /opt/sbin/cryptsetup
        libcryptsetup.so.4 => /opt/lib/libcryptsetup.so.4 (0x40107000)
        libpopt.so.0 => /opt/lib/libpopt.so.0 (0x400d1000)
        libgcc_s.so.1 => /opt/lib/libgcc_s.so.1 (0x400eb000)
        libc.so.6 => /opt/lib/libc.so.6 (0x40137000)
        libuuid.so.1 => /opt/lib/libuuid.so.1 (0x40273000)
        libdevmapper.so.1.02 => /opt/lib/libdevmapper.so.1.02 (0x40286000)
        libgcrypt.so.20 => /opt/lib/libgcrypt.so.20 (0x402f9000)
        libgpg-error.so.0 => /opt/lib/libgpg-error.so.0 (0x40020000)
        /opt/lib/ld-linux.so.3 (0x40054000)
        libm.so.6 => /opt/lib/libm.so.6 (0x403a2000)
ldd /mmc/sbin/cryptsetup
        libcryptsetup.so.4 => /mmc/lib/libcryptsetup.so.4 (0x4015d000)
        libuuid.so.1 => /mmc/lib/libuuid.so.1 (0x4018e000)
        libdevmapper.so.1.02 => /mmc/lib/libdevmapper.so.1.02 (0x400db000)
        libgcrypt.so.20 => /mmc/lib/libgcrypt.so.20 (0x401a1000)
        libgpg-error.so.0 => /mmc/lib/libgpg-error.so.0 (0x40137000)
        libpopt.so.0 => /mmc/lib/libpopt.so.0 (0x40267000)
        libc.so.0 => /mmc/lib/libc.so.0 (0x40281000)
        ld-uClibc.so.1 => /mmc/lib/ld-uClibc.so.1 (0x400bc000)

Then, I ran my dependency walker against both programs. Not sure if this is useful. The segmentation fault in the dynamically-linked version of cryptsetup on tomatoware happens after the normal 2 minute wait. This tells me that the dynamic linking issue is probably related to libdevmapper, simply because the entry in /dev/mapper that normally gets created, is created after the 2 minute wait, and it is not creating this entry when the segmentation fault occurs.

/opt/sbin/cryptsetup
├─/opt/lib/libcryptsetup.so.4
│ ├─/opt/lib/libuuid.so.1
│ │ └─/opt/lib/libc.so.6
│ │   └─/opt/lib/ld-linux.so.3
│ ├─/opt/lib/libdevmapper.so.1.02
│ │ ├─/opt/lib/libuuid.so.1
│ │ │ └─/opt/lib/libc.so.6
│ │ │   └─/opt/lib/ld-linux.so.3
│ │ ├─/opt/lib/libm.so.6
│ │ │ ├─/opt/lib/libc.so.6
│ │ │ │ └─/opt/lib/ld-linux.so.3
│ │ │ └─/opt/lib/ld-linux.so.3
│ │ └─/opt/lib/libc.so.6
│ │   └─/opt/lib/ld-linux.so.3
│ ├─/opt/lib/libgcrypt.so.20
│ │ ├─/opt/lib/libgpg-error.so.0
│ │ │ └─/opt/lib/libc.so.6
│ │ │   └─/opt/lib/ld-linux.so.3
│ │ └─/opt/lib/libc.so.6
│ │   └─/opt/lib/ld-linux.so.3
│ ├─/opt/lib/libgpg-error.so.0
│ │ └─/opt/lib/libc.so.6
│ │   └─/opt/lib/ld-linux.so.3
│ ├─/opt/lib/libgcc_s.so.1
│ │ └─/opt/lib/libc.so.6
│ │   └─/opt/lib/ld-linux.so.3
│ └─/opt/lib/libc.so.6
│   └─/opt/lib/ld-linux.so.3
├─/opt/lib/libpopt.so.0
│ └─/opt/lib/libc.so.6
│   └─/opt/lib/ld-linux.so.3
├─/opt/lib/libgcc_s.so.1
│ └─/opt/lib/libc.so.6
│   └─/opt/lib/ld-linux.so.3
└─/opt/lib/libc.so.6
  └─/opt/lib/ld-linux.so.3
/mmc/sbin/cryptsetup
├─/mmc/lib/libcryptsetup.so.4
│ ├─/mmc/lib/libuuid.so.1
│ │ ├─/mmc/lib/libc.so.0
│ │ │ └─/mmc/lib/ld-uClibc.so.1
│ │ └─/mmc/lib/ld-uClibc.so.1
│ ├─/mmc/lib/libdevmapper.so.1.02
│ │ └─/mmc/lib/libc.so.0
│ │   └─/mmc/lib/ld-uClibc.so.1
│ ├─/mmc/lib/libgcrypt.so.20
│ │ ├─/mmc/lib/libgpg-error.so.0
│ │ │ └─/mmc/lib/libc.so.0
│ │ │   └─/mmc/lib/ld-uClibc.so.1
│ │ └─/mmc/lib/libc.so.0
│ │   └─/mmc/lib/ld-uClibc.so.1
│ ├─/mmc/lib/libgpg-error.so.0
│ │ └─/mmc/lib/libc.so.0
│ │   └─/mmc/lib/ld-uClibc.so.1
│ ├─/mmc/lib/libpopt.so.0
│ │ └─/mmc/lib/libc.so.0
│ │   └─/mmc/lib/ld-uClibc.so.1
│ └─/mmc/lib/libc.so.0
│   └─/mmc/lib/ld-uClibc.so.1
├─/mmc/lib/libuuid.so.1
│ ├─/mmc/lib/libc.so.0
│ │ └─/mmc/lib/ld-uClibc.so.1
│ └─/mmc/lib/ld-uClibc.so.1
├─/mmc/lib/libdevmapper.so.1.02
│ └─/mmc/lib/libc.so.0
│   └─/mmc/lib/ld-uClibc.so.1
├─/mmc/lib/libgcrypt.so.20
│ ├─/mmc/lib/libgpg-error.so.0
│ │ └─/mmc/lib/libc.so.0
│ │   └─/mmc/lib/ld-uClibc.so.1
│ └─/mmc/lib/libc.so.0
│   └─/mmc/lib/ld-uClibc.so.1
├─/mmc/lib/libgpg-error.so.0
│ └─/mmc/lib/libc.so.0
│   └─/mmc/lib/ld-uClibc.so.1
├─/mmc/lib/libpopt.so.0
│ └─/mmc/lib/libc.so.0
│   └─/mmc/lib/ld-uClibc.so.1
└─/mmc/lib/libc.so.0
  └─/mmc/lib/ld-uClibc.so.1
lancethepants commented 7 years ago

I always like to check out how other people are compiling things, so I usually check out the buildroot repo. Here's is how buildroot builds lvm2 if you want to mess around with some of their settings. https://git.busybox.net/buildroot/tree/package/lvm2

Supposedly though don't build a static lvm2, cause they say "It fails to build statically". Interesting.

Your experience is also interesting because it's been exactly the opposite of my experiences regarding static vs dyanmic. Whever I have a segfault, it always has been an issue of the static binary, but if I compile it dynamically, it works just fine. I just made a new release of tomatoware. I don't know if it will change anything, but maybe worth a try. It now has gcc 6.3, if you weren't running nighty builds already.

blackfuel commented 7 years ago

Last I remember, the Cryptsetup program requires another program from LVM2 called dmsetup. When I run my tests again, must pay attention that it's using the correct dmsetup from tomatoware and not Entware-ng. I think I had put a firmware link to the Entware-ng version in /sbin, which is searched first.

I'm finally able to compile Cryptsetup for AsusWRT (w/ Broadcom ARM toolchain). The tomatoware cryptsetup.static is still faster for unlocking a Veracrypt file container (2 minutes), compared to my Cryptsetup/AsusWRT build (5 minutes), for the crypto backends openssl and gcrypt. This I don't understand. Maybe it's some compiler optimization in the tomatoware buildroot? https://github.com/blackfuel/cryptsetup-arm-asuswrt

I found there's excellent performance gains when using the crypto backend, nettle. It unlocked my Veracrypt file container in 90 seconds. I was using my Cryptsetup/AsusWRT build.

lancethepants commented 7 years ago

Have you tried tomatoware with the nettle backend yet? Nettle is already included in tomatoware.

blackfuel commented 7 years ago

Tomatoware cryptsetup.static with nettle crypto backend took 70 seconds to unlock my Veracrypt file container on the RT-AC68U at 1200MHz. That's the fastest.

lancethepants commented 7 years ago

The tomatoware toolchain uses significantly newer versions of uclibc-ng, binutils, & gcc, so this is likely why it is faster.