Adding ARM Architecture

[decidedlygray]

So, I've been fighting with this for a few days. I'm not sure if this project is still being supported, but I thought I'd take a shot and try and see if you might know what is going wrong here.

I am trying to add ARM support. The included buildmips.sh and buildmipsel.sh work great, so I installed the ARM cross compile tools and set up a similar buildarm.sh script:

#!/bin/sh

export ARCH=arm
TARGET=$1

# Sets up toolchain environment variables for various mips toolchain

warn()
{
    echo "$1" >&2
}

if [ ! -z $(which arm-linux-gnueabi-gcc-5) ];
then
    export CC=$(which arm-linux-gnueabi-gcc-5)
else
    warn "Not setting CC: can't locate arm-linux-gnueabi-gcc-5."
fi

if [ ! -z $(which arm-linux-gnueabi-ld) ];
then
    export LD=$(which arm-linux-gnueabi-ld)
else
    warn "Not setting LD: can't locate arm-linux-gnueabi-ld."
fi

if [ ! -z $(which arm-linux-gnueabi-ar) ];
then
    export AR=$(which arm-linux-gnueabi-ar)
else
    warn "Not setting AR: can't locate arm-linux-gnueabi-ar."
fi

if [ ! -z $(which arm-linux-gnueabi-strip) ];
then
    export STRIP=$(which arm-linux-gnueabi-strip)
else
    warn "Not setting STRIP: can't locate arm-linux-gnueabi-strip."
fi

if [ ! -z $(which arm-linux-gnueabi-nm) ];
then
    export NM=$(which arm-linux-gnueabi-nm)
else
    warn "Not setting NM: can't lcoate arm-linux-gnueabi-nm."
fi

make $TARGET || exit $?

Side note: arm-linux-gnueabi-gcc-5 is being used instead of arm-linux-gnueabi-gcc because, using some code from the devtty0's blog to build an nvram hook, I found it was looking for libc.so.6 instead of the device's libc.so.0. I also tried to build against the devices libc.so.0 but was quickly out of my depth

Everything compiles fine, but when I execute:

sudo chroot . ./qemu-arm-static -E LD_PRELOAD="/libnvram-faker.so" sbin/version

I get the following error:

sbin/version: symbol 'initialize_ini': can't resolve symbol

sbin/version: symbol 'end': can't resolve symbol

Both of these functions obviously exist in nvram-faker/nvram-faker.c, I don't understand :(

[zcutlip]

Do you mind sending me a tarball of your copy of nvram-faker, including all built binaries and intermediate object files? It's been a while since I worked with this project but I'll take a look.

[decidedlygray]

Sure thing: https://mega.nz/#!t3ZBjbRQ!rsMYM2pywvteIKaWqTmwRslEKYvMZzfOSNWb9vfwI2c

Also, here is the output of file, which indicates that its not stripped: libnvram-faker.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=3885891187d592b86ee75ef5c5e174f577ab1e0c, not stripped

[zcutlip]

For reasons I can't figure out, I can't get the library to preload on my raspberry pi, even one that I built myself. However, in the process of trying, I learned about LD_DEBUG. You can set it =all for all debug information, or =help to see debugging options. This is true at least for ld-linux.so. Not sure about uclibc or others.

Can you try setting the 'LD_DEBUG=all' environment variable in addition to the LD_PRELOAD, and pasting the output when you run your executable?

[decidedlygray]

So, I was playing with this a bit, and read up on it here http://www.bnikolic.co.uk/blog/linux-ld-debug.html. No matter which options I try (help, all, libs) I do not see any different output than the same:

sbin/version: symbol 'initialize_ini': can't resolve symbol

sbin/version: symbol 'end': can't resolve symbol

I also tried different binaries on the target firmware, but nothing changes.

One thing I will note is that in recreating this all again, I had to do the following to get libnvram-faker.so to even load:

cp /usr/arm-linux-gnueabi/lib/ld-linux.so.3 lib/

Where lib/ is the lib directory that ends up in my chroot'ed environment. If I don't do that I get:

sbin/version: can't load library 'ld-linux.so.3'

Because libnvram-faker.so wants ld-linux.so.3

In case it helps at all, here is the strace from qemu: Command -

 sudo chroot . ./qemu-arm-static -strace -E LD_PRELOAD="/libnvram-faker.so" sbin/version

Output -

3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf67f0000
3911 open("/libnvram-faker.so",O_RDONLY) = 3
3911 fstat(3,0xf6fff3d8) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf67ef000
3911 read(3,0xf67ef000,4096) = 4096
3911 mmap2(NULL,77824,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf67dc000
3911 mmap2(0xf67dc000,5736,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf67dc000
3911 mmap2(0xf67ed000,4220,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x1) = 0xf67ed000
3911 close(3) = 0
3911 munmap(0xf67ef000,4096) = 0
3911 open("/lib/libnvram.so",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb80) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf67db000
3911 read(3,0xf67db000,4096) = 4096
3911 mmap2(NULL,516096,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf675d000
3911 mmap2(0xf675d000,53064,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf675d000
3911 mmap2(0xf6772000,19020,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xd) = 0xf6772000
3911 mmap2(0xf6777000,408148,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xf6777000
3911 close(3) = 0
3911 munmap(0xf67db000,4096) = 0
3911 open("/lib/libacos_shared.so",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/lib/libacos_shared.so",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/usr/lib/libacos_shared.so",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb58) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf675c000
3911 read(3,0xf675c000,4096) = 4096
3911 mmap2(NULL,311296,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf6710000
3911 mmap2(0xf6710000,224940,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf6710000
3911 mmap2(0xf674f000,19164,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x37) = 0xf674f000
3911 mmap2(0xf6754000,31012,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xf6754000
3911 close(3) = 0
3911 munmap(0xf675c000,4096) = 0
3911 open("/lib/libgcc_s.so.1",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb60) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf670f000
3911 read(3,0xf670f000,4096) = 4096
3911 mmap2(NULL,73728,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf66fd000
3911 mmap2(0xf66fd000,39128,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf66fd000
3911 mmap2(0xf670e000,2800,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x9) = 0xf670e000
3911 close(3) = 0
3911 munmap(0xf670f000,4096) = 0
3911 open("/lib/libnat.so",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/lib/libnat.so",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/usr/lib/libnat.so",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb38) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf66fc000
3911 read(3,0xf66fc000,4096) = 4096
3911 mmap2(NULL,98304,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf66e4000
3911 mmap2(0xf66e4000,64224,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf66e4000
3911 mmap2(0xf66fb000,3620,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xf) = 0xf66fb000
3911 close(3) = 0
3911 munmap(0xf66fc000,4096) = 0
3911 open("/lib/libc.so.0",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb40) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf66e3000
3911 read(3,0xf66e3000,4096) = 4096
3911 mmap2(NULL,475136,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf666f000
3911 mmap2(0xf666f000,413376,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf666f000
3911 mmap2(0xf66dc000,4980,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x65) = 0xf66dc000
3911 mmap2(0xf66de000,17872,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xf66de000
3911 close(3) = 0
3911 munmap(0xf66e3000,4096) = 0
3911 open("/lib/libc.so.6",O_RDONLY) = 3
3911 fstat(3,0xf6ffeb30) = 0
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_UNINITIALIZED,-1,0) = 0xf666e000
3911 read(3,0xf666e000,4096) = 4096
3911 mmap2(NULL,475136,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf65fa000
3911 mmap2(0xf65fa000,413376,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_FIXED,3,0) = 0xf65fa000
3911 mmap2(0xf6667000,4980,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x65) = 0xf6667000
3911 mmap2(0xf6669000,17872,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xf6669000
3911 close(3) = 0
3911 munmap(0xf666e000,4096) = 0
3911 open("/lib/ld-linux.so.3",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/lib/ld-linux.so.3",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/usr/lib/ld-linux.so.3",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 open("/usr/X11R6/lib/ld-linux.so.3",O_RDONLY) = -1 errno=2 (No such file or directory)
3911 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xf65f9000
3911 write(2,0xf65f9000,0) = 0
3911 write(2,0xf6fff8a5,12)sbin/version = 12
3911 write(2,0xf65f9002,22): can't load library ' = 22
3911 write(2,0xf67dc52c,13)ld-linux.so.3 = 13
3911 write(2,0xf65f901a,2)'
 = 2
3911 munmap(0xf65f9000,4096) = 0
3911 exit(16)

[decidedlygray]

Realizing now that the need for ld-linux.so.3 means glibc, and I see ld-uClibc.so.0 in the lib/ directory on my target, so I'm wondering if my toolchain is is wrong.

[decidedlygray]

Ok, so that was silly. Wrong toolchain.

I can "successfully" (in quotes because I still actually need to figure out what nvram values to supply) execute the following:

sudo chroot . ./qemu-arm-static -E LD_PRELOAD="/libnvram-faker.so" sbin/version

My updated buildarm.sh, now named a much more specific buildarm_armv7-eabihf-glibc.sh:

#!/bin/sh

export ARCH=arm
TARGET=$1

# Sets up toolchain environment variables for various mips toolchain

warn()
{
    echo "$1" >&2
}

if [ ! -z $(which arm-linux-gcc-5.4.0) ];
then
    export CC=$(which arm-linux-gcc-5.4.0)
else
    warn "Not setting CC: can't locate arm-linux-gcc-5.4.0."
fi

if [ ! -z $(which arm-linux-ld) ];
then
    export LD=$(which arm-linux-ld)
else
    warn "Not setting LD: can't locate arm-linux-ld."
fi

if [ ! -z $(which arm-linux-ar) ];
then
    export AR=$(which arm-linux-ar)
else
    warn "Not setting AR: can't locate arm-linux-ar."
fi

if [ ! -z $(which arm-linux-strip) ];
then
    export STRIP=$(which arm-linux-strip)
else
    warn "Not setting STRIP: can't locate arm-linux-strip."
fi

if [ ! -z $(which arm-linux-nm) ];
then
    export NM=$(which arm-linux-nm)
else
    warn "Not setting NM: can't lcoate arm-linux-nm."
fi

make $TARGET || exit $?

After obtaining the proper toolchain from http://toolchains.free-electrons.com/ (armv7-eabihf, glibc), and setting my path to pick up the binaries, it looks like I'm finally on the right track.

Thank you for your help on this!

Now if only there was some way to make ioctl-faker...

[zcutlip]

So here's a little background if it helps (if you're already familiar, I apologize; don't mean to patronize). On linux and other unix-ey OSes, ld.whatever.so is actually not a shared library at all, even though it ends in ".so". In fact you can run it on its own. Try it:

$  /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
...

This is actually the loader. It's an elf executable that mmaps your executable into memory, then parses all of its headers and figures out what additional libraries to map into memory (and parses those libraries' headers and maps their dependencies into memory, etc.). Then once it's done all that, it hands off control to the image of your executable that it mapped into memory.

So when you build an executable and it's "linked" against that loader, what has happened is the toolchain that built it has said "this is the loader that knows how to parse this executable file format. other loaders probably won't understand it." So as you have discovered, this means the compiler toolchain is tightly coupled to the OS's loader and C runtime libraries.

I'm really glad you got it working. Can I close this issue now? If so, you should send me a pull request, so I can build with arm toolchains.

[decidedlygray]

I was vaguely aware of it's function, and your explanation helped clarify a lot. I also didn't know you could run it on its own. IoT exploration is certainly helping me fill in my gaps of Linux knowledge.

You can close the issue, I will get a PR together tomorrow. Thanks again!

[0xsmirk]

I always can't build the armhf ! Can you prived the armhf!

[zcutlip]

should be fixed in PR #4