Build failure on Ubuntu 16.04.1

[kinow]

Hi,

followed instructions on README, and almost everything worked OK. cmake .. failed with:

CMake Warning at deps/cmake_modules/CodeCoverage.cmake:116 (MESSAGE):
  Code coverage results with an optimized (non-Debug) build may be misleading
Call Stack (most recent call first):
  CMakeLists.txt:20 (INCLUDE)

CMake Error at CMakeLists.txt:314 (target_link_libraries):
  Cannot specify link libraries for target "nxemu" which is not built by this
  project.

Adding the following in the elseif(${CMAKE_SYSTEM_NAME} MATCHES "Linux"):

add_library(nxemu SHARED src/emu/emu.c) # ok to not include  src/emu/emu-macos.c I think?

Makes cmake happy, but make fails with:

CMakeFiles/nxemu.dir/src/emu/emu.c.o: In function `init_program_ctx':
emu.c:(.text+0x3537): undefined reference to `init_header'
CMakeFiles/nxemu.dir/src/emu/emu.c.o: In function `parse_macho_fp':
emu.c:(.text+0x35e8): undefined reference to `get_magic_number'
/usr/bin/ld: libnxemu.so: hidden symbol `get_magic_number' isn't defined
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
CMakeFiles/nxemu.dir/build.make:100: recipe for target 'libnxemu.so' failed
make[2]: *** [libnxemu.so] Error 1
CMakeFiles/Makefile2:991: recipe for target 'CMakeFiles/nxemu.dir/all' failed
make[1]: *** [CMakeFiles/nxemu.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

Any idea how I could get the project to build on Linux again? Trying to reproduce the issue #7 :-)

Thanks

[hbowden]

So the problem is I started writing a X86-64 emulator for nextgen so I could start building something similar to Microsoft's SAGE fuzzer: http://research.microsoft.com/en-us/um/people/pg/public_psfiles/SAGE-in-one-slide.pdf, However It currently expects Mach-O instead of the ELF file format found on FreeBSD and Linux. The undefined function references are implemented in the file emu-macos.c. My plan is to abstract away the Mach-O specific code and implement the ELF specific functions for Linux and FreeBSD. I will hopefully have that done tonight and I will push it to the master branch. As a quick workaround remove all the places in the CMakeList.txt where the emu library is being built, you shouldn't need it to build the rest of nextgen. Below is the lines I believe need to be removed.

target_link_libraries(nxemu nxio)
target_link_libraries(nxemu nxconcurrent)
target_link_libraries(nxemu nxmemory)

[kinow]

Yup, removing these three lines the build worked for me on Linux.

I think we can close the issue as this will be fixed anyway with your work on the Mach-O abstraction layer. Is that OK?

[kinow]

Just adding more info since I have a build passing with cmake .. and make. Here's the output of make test

kinow@localhost:~/Development/cpp/workspace/nextgen/build$ make test
Running tests...
Test project /home/kinow/Development/cpp/workspace/nextgen/build
    Start 1: emu-integration-test
Could not find executable emu-integration-test
Looked in the following places:
emu-integration-test
emu-integration-test
Release/emu-integration-test
Release/emu-integration-test
Debug/emu-integration-test
Debug/emu-integration-test
MinSizeRel/emu-integration-test
MinSizeRel/emu-integration-test
RelWithDebInfo/emu-integration-test
RelWithDebInfo/emu-integration-test
Deployment/emu-integration-test
Deployment/emu-integration-test
Development/emu-integration-test
Development/emu-integration-test
Unable to find executable: emu-integration-test
1/9 Test #1: emu-integration-test .............***Not Run   0.00 sec
    Start 2: emu-unit-test
Could not find executable emu-unit-test
Looked in the following places:
emu-unit-test
emu-unit-test
Release/emu-unit-test
Release/emu-unit-test
Debug/emu-unit-test
Debug/emu-unit-test
MinSizeRel/emu-unit-test
MinSizeRel/emu-unit-test
RelWithDebInfo/emu-unit-test
RelWithDebInfo/emu-unit-test
Deployment/emu-unit-test
Deployment/emu-unit-test
Development/emu-unit-test
Development/emu-unit-test
Unable to find executable: emu-unit-test
2/9 Test #2: emu-unit-test ....................***Not Run   0.00 sec
    Start 3: resource-integration-test
Could not find executable resource-integration-test
Looked in the following places:
resource-integration-test
resource-integration-test
Release/resource-integration-test
Release/resource-integration-test
Debug/resource-integration-test
Debug/resource-integration-test
MinSizeRel/resource-integration-test
MinSizeRel/resource-integration-test
RelWithDebInfo/resource-integration-test
RelWithDebInfo/resource-integration-test
Deployment/resource-integration-test
Deployment/resource-integration-test
Development/resource-integration-test
Development/resource-integration-test
Unable to find executable: resource-integration-test
3/9 Test #3: resource-integration-test ........***Not Run   0.00 sec
    Start 4: runtime-integration-test
Could not find executable runtime-integration-test
Looked in the following places:
runtime-integration-test
runtime-integration-test
Release/runtime-integration-test
Release/runtime-integration-test
Debug/runtime-integration-test
Debug/runtime-integration-test
MinSizeRel/runtime-integration-test
MinSizeRel/runtime-integration-test
RelWithDebInfo/runtime-integration-test
RelWithDebInfo/runtime-integration-test
Deployment/runtime-integration-test
Deployment/runtime-integration-test
Development/runtime-integration-test
Development/runtime-integration-test
Unable to find executable: runtime-integration-test
4/9 Test #4: runtime-integration-test .........***Not Run   0.00 sec
    Start 5: utils-unit-test
Could not find executable utils-unit-test
Looked in the following places:
utils-unit-test
utils-unit-test
Release/utils-unit-test
Release/utils-unit-test
Debug/utils-unit-test
Debug/utils-unit-test
MinSizeRel/utils-unit-test
MinSizeRel/utils-unit-test
RelWithDebInfo/utils-unit-test
RelWithDebInfo/utils-unit-test
Deployment/utils-unit-test
Deployment/utils-unit-test
Development/utils-unit-test
Development/utils-unit-test
Unable to find executable: utils-unit-test
5/9 Test #5: utils-unit-test ..................***Not Run   0.00 sec
    Start 6: memory-unit-test
Could not find executable memory-unit-test
Looked in the following places:
memory-unit-test
memory-unit-test
Release/memory-unit-test
Release/memory-unit-test
Debug/memory-unit-test
Debug/memory-unit-test
MinSizeRel/memory-unit-test
MinSizeRel/memory-unit-test
RelWithDebInfo/memory-unit-test
RelWithDebInfo/memory-unit-test
Deployment/memory-unit-test
Deployment/memory-unit-test
Development/memory-unit-test
Development/memory-unit-test
Unable to find executable: memory-unit-test
6/9 Test #6: memory-unit-test .................***Not Run   0.00 sec
    Start 7: memory-intergration-test
Could not find executable memory-intergration-test
Looked in the following places:
memory-intergration-test
memory-intergration-test
Release/memory-intergration-test
Release/memory-intergration-test
Debug/memory-intergration-test
Debug/memory-intergration-test
MinSizeRel/memory-intergration-test
MinSizeRel/memory-intergration-test
RelWithDebInfo/memory-intergration-test
RelWithDebInfo/memory-intergration-test
Deployment/memory-intergration-test
Deployment/memory-intergration-test
Development/memory-intergration-test
Development/memory-intergration-test
Unable to find executable: memory-intergration-test
7/9 Test #7: memory-intergration-test .........***Not Run   0.00 sec
    Start 8: crypto-unit-test
Could not find executable crypto-unit-test
Looked in the following places:
crypto-unit-test
crypto-unit-test
Release/crypto-unit-test
Release/crypto-unit-test
Debug/crypto-unit-test
Debug/crypto-unit-test
MinSizeRel/crypto-unit-test
MinSizeRel/crypto-unit-test
RelWithDebInfo/crypto-unit-test
RelWithDebInfo/crypto-unit-test
Deployment/crypto-unit-test
Deployment/crypto-unit-test
Development/crypto-unit-test
Development/crypto-unit-test
Unable to find executable: crypto-unit-test
8/9 Test #8: crypto-unit-test .................***Not Run   0.00 sec
    Start 9: concurrent-unit-test
Could not find executable concurrent-unit-test
Looked in the following places:
concurrent-unit-test
concurrent-unit-test
Release/concurrent-unit-test
Release/concurrent-unit-test
Debug/concurrent-unit-test
Debug/concurrent-unit-test
MinSizeRel/concurrent-unit-test
MinSizeRel/concurrent-unit-test
RelWithDebInfo/concurrent-unit-test
RelWithDebInfo/concurrent-unit-test
Deployment/concurrent-unit-test
Deployment/concurrent-unit-test
Development/concurrent-unit-test
Development/concurrent-unit-test
Unable to find executable: concurrent-unit-test
9/9 Test #9: concurrent-unit-test .............***Not Run   0.00 sec

0% tests passed, 9 tests failed out of 9

Total Test time (real) =   0.00 sec

The following tests FAILED:
      1 - emu-integration-test (Not Run)
      2 - emu-unit-test (Not Run)
      3 - resource-integration-test (Not Run)
      4 - runtime-integration-test (Not Run)
      5 - utils-unit-test (Not Run)
      6 - memory-unit-test (Not Run)
      7 - memory-intergration-test (Not Run)
      8 - crypto-unit-test (Not Run)
      9 - concurrent-unit-test (Not Run)
Errors while running CTest
Makefile:61: recipe for target 'test' failed
make: *** [test] Error 8

[kinow]

make coverage has a similar output, but I'm also not able to execute the generated nextgen command. There are some missing libraries, which I will start investigating if there's anything missing in my local environment.

Here's the output of ldd nextgen:

kinow@localhost:~/Development/cpp/workspace/nextgen/build$ ldd nextgen
    linux-vdso.so.1 =>  (0x00007ffcbacd1000)
    libnxruntime.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxruntime.so (0x00007f0c33126000)
    libnxutils.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxutils.so (0x00007f0c32f1b000)
    libnxmemory.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxmemory.so (0x00007f0c32d12000)
    libnxio.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxio.so (0x00007f0c32b0f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0c3270b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0c324ee000)
    libck.so.0 => not found
    libnxdisas.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxdisas.so (0x00007f0c322e4000)
    libnxfile.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxfile.so (0x00007f0c320da000)
    libnxgenetic.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxgenetic.so (0x00007f0c31ed0000)
    libnxsyscall.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxsyscall.so (0x00007f0c31c9f000)
    libnxconcurrent.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxconcurrent.so (0x00007f0c31a90000)
    libnxprobe.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxprobe.so (0x00007f0c31888000)
    libnxresource.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxresource.so (0x00007f0c3167c000)
    libnxlog.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxlog.so (0x00007f0c3146d000)
    libnxplugin.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxplugin.so (0x00007f0c31264000)
    libnxcrypto.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxcrypto.so (0x00007f0c31059000)
    /lib64/ld-linux-x86-64.so.2 (0x000055a97d81f000)
    libcapstone.so.3 => not found
    libnxmutate.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxmutate.so (0x00007f0c30e55000)
    libnxnetwork.so => /home/kinow/Development/cpp/workspace/nextgen/build/libnxnetwork.so (0x00007f0c30c4c000)
    libck.so.0 => not found
    libsqlite3.so => /home/kinow/Development/cpp/workspace/nextgen/deps/sqlite/libsqlite3.so (0x00007f0c3097c000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0c30777000)
    libcrypto.so.38 => /home/kinow/Development/cpp/workspace/nextgen/deps/libressl-2.5.0/crypto/.libs/libcrypto.so.38 (0x00007f0c303a0000)

[kinow]

Solved missing libraries with sudo apt install libck-dev libcapstone-dev -y

Running nextgen though outputs

kinow@localhost:~/Development/cpp/workspace/nextgen/build$ ./nextgen
Can't parse command line.
kinow@localhost:~/Development/cpp/workspace/nextgen/build$ ./nextgen -h
./nextgen: invalid option -- 'h'
Unknown option
Can't parse command line.
kinow@localhost:~/Development/cpp/workspace/nextgen/build$ ./nextgen --help
Can't parse command line.
kinow@localhost:~/Development/cpp/workspace/nextgen/build$ ./nextgen --h
Can't parse command line.

Probably related to #13

[hbowden]

When you built nextgen did you run sudo make install after building it?

[kinow]

No, I did not. I used the nextgen binary in the build directory. Should I give it a try and run sudo make install?

[hbowden]

Yea that should install the needed libraries that you installed with apt.

[kinow]

Gotcha. Will run it tonight at home and update the issue :+1:

[hbowden]

I also noticed that you used make test, try make check instead. make check will build the test suite and then run it. make test will just try to run the test suite but will fail if the test suit has not been built.

[kinow]

sudo make install works all right. No errors, and I can open a new terminal and nextgen is included in the path (more specifically, in /usr/local/bin/nextgen).

make check failed with:

kinow@localhost:~/Development/cpp/workspace/nextgen/build$ make check
[  2%] Built target nxio
[  5%] Built target nxmemory
[  9%] Built target nxcrypto
[ 13%] Built target nxutils
[ 18%] Built target nxconcurrent
Scanning dependencies of target memory-unit-test
[ 19%] Building C object CMakeFiles/memory-unit-test.dir/tests/memory/unit/tests.c.o
[ 20%] Building C object CMakeFiles/memory-unit-test.dir/deps/Unity/src/unity.c.o
[ 21%] Linking C executable memory-unit-test
[ 21%] Built target memory-unit-test
Scanning dependencies of target emu-unit-test
[ 22%] Building C object CMakeFiles/emu-unit-test.dir/tests/emu/unit/tests.c.o
In file included from /home/kinow/Development/cpp/workspace/nextgen/tests/emu/unit/tests.c:20:0:
/home/kinow/Development/cpp/workspace/nextgen/tests/emu/unit/../../../src/emu/emu-macos.c:20:27: fatal error: mach-o/loader.h: No such file or directory
compilation terminated.
CMakeFiles/emu-unit-test.dir/build.make:62: recipe for target 'CMakeFiles/emu-unit-test.dir/tests/emu/unit/tests.c.o' failed
make[3]: *** [CMakeFiles/emu-unit-test.dir/tests/emu/unit/tests.c.o] Error 1
CMakeFiles/Makefile2:154: recipe for target 'CMakeFiles/emu-unit-test.dir/all' failed
make[2]: *** [CMakeFiles/emu-unit-test.dir/all] Error 2
CMakeFiles/Makefile2:1263: recipe for target 'CMakeFiles/check.dir/rule' failed
make[1]: *** [CMakeFiles/check.dir/rule] Error 2
Makefile:567: recipe for target 'check' failed
make: *** [check] Error 2

I think I may have to disable loading some library in the tests too?

[hbowden]

Try removing the lines below from the cmake config so make check does not try and run the emulator test.

add_executable(emu-unit-test EXCLUDE_FROM_ALL tests/emu/unit/tests.c deps/${UNITY}/src/unity.c)
target_link_libraries(emu-unit-test nxmemory)
target_link_libraries(emu-unit-test nxio)
target_link_libraries(emu-unit-test nxconcurrent)

add_executable(emu-integration-test EXCLUDE_FROM_ALL tests/emu/integration/tests.c deps/${UNITY}/src/unity.c)
target_link_libraries(emu-integration-test nxmemory)
target_link_libraries(emu-integration-test nxio)
target_link_libraries(emu-integration-test nxconcurrent)

add_sanitizers(emu-integration-test)
add_sanitizers(emu-unit-test)

add_test(emu-integration-test emu-integration-test)
add_test(emu-unit-test emu-unit-test)

add_dependencies(check emu-integration-test)
add_dependencies(check emu-unit-test)