make test_dependencies failed

[adahsuzixin]

Hi, team, I met problem when running test.

I tried to add an instruction VINSERTF128,

diff --git a/remill/Arch/X86/Semantics/AVX.cpp b/remill/Arch/X86/Semantics/AVX.cpp
index f7069a0..ad73343 100644
--- a/remill/Arch/X86/Semantics/AVX.cpp
+++ b/remill/Arch/X86/Semantics/AVX.cpp
@@ -51,4 +51,21 @@ DEF_SEM(VPBROADCASTB, D dst, S1 src1) {
 DEF_ISEL(VZEROUPPER) = DoVZEROUPPER;
 DEF_ISEL(VPBROADCASTB_YMMqq_XMMb) = VPBROADCASTB<VV256W, V128, 32, 256>;

+namespace {
+
+template<typename S2>
+DEF_SEM(VINSERTF128, VV256W dst, V256 src1, S2 src2, I8 src3) {
+  auto dst_vec = UReadV128(src1);
+  auto src2_vec = UReadV128(src2);
+  auto src3_i8 = Read(src3);
+  std::size_t i = (src3_i8 != 0) ? 1 : 0;
+  dst_vec  = UInsertV128(dst_vec, i, UExtractV128(src2_vec, 0));
+  UWriteV128(dst, dst_vec);
+  return memory;
+}
+
+}  // namespace
+DEF_ISEL(VINSERTF128_YMMqq_YMMqq_MEMdq_IMMb) = VINSERTF128<MV128>;
+DEF_ISEL(VINSERTF128_YMMqq_YMMqq_XMMdq_IMMb) = VINSERTF128<V128>;

diff --git a/tests/X86/Tests.S b/tests/X86/Tests.S
index 1a9d82f..848cab7 100644
--- a/tests/X86/Tests.S
+++ b/tests/X86/Tests.S
@@ -361,6 +361,7 @@ SYMBOL(__x86_test_table_begin):

 #include "tests/X86/AVX/VZEROUPPER.S"
 #include "tests/X86/AVX/VPBROADCASTB.S"
+/* #include "tests/X86/AVX/VINSERTF128.S" */

This is the testcase I used to test this instruction.

/* VINSERTF128.S */
#if HAS_FEATURE_AVX

TEST_BEGIN(VINSERTF128, 3)
TEST_INPUTS(
    0x7ff8000012345678,0x0123456789abcdef,0xfedcba9876543210,
    0x7ff8000012345678,0x1111222233334444,0x5555666677778888
)
    push 0;
    push ARG2_64;
    push 0;
    push ARG3_64;
    vmovdqu ymm2, [rsp];
    movq xmm3, ARG1_64;
    vinsertf128 ymm1, ymm2, xmm3, 1;
TEST_END

#endif  // HAS_FEATURE_AVX

When run make test_dependencies and running lift-x86_avx-tests, the problem occur.

cd /home/suzixin/initCode/remill/scripts/remill-build/tests/X86 && ./lift-x86_avx-tests --arch x86_avx --bc_out tests_x86_avx.bc
F20200628 20:26:49.251901 64622 Arch.cpp:991] Check failed: parent_indexes.size() < reg.gep_index_list.size() (3 vs. 3) Parent register arrayidx index list is same size as sub-register arrayidx137 index list
*** Check failure stack trace: ***
    @          0x17001cc  google::LogMessageFatal::~LogMessageFatal()
    @           0xa60b24  remill::Arch::CollectRegisters()
make[3]: *** [tests/X86/CMakeFiles/run-x86_avx-tests.dir/build.make:89: tests/X86/tests_x86_avx.bc] Aborted (core dumped)
make[3]: Leaving directory '/home/suzixin/initCode/remill/scripts/remill-build'
make[2]: *** [CMakeFiles/Makefile2:1895: tests/X86/CMakeFiles/run-x86_avx-tests.dir/all] Error 2
make[2]: Leaving directory '/home/suzixin/initCode/remill/scripts/remill-build'
make[1]: *** [CMakeFiles/Makefile2:1244: CMakeFiles/test_dependencies.dir/rule] Error 2
make[1]: Leaving directory '/home/suzixin/initCode/remill/scripts/remill-build'
make: *** [Makefile:627: test_dependencies] Error 2

Even if I undo the modification to Tests.S, the problem still appears, could you please help me to find the root cause?

[pgoodman]

In the immediate/short term, can you try using the api_improvements branch?

Ultimately this stems from a seemingly good idea that time has proven to be a bad idea: deriving all register names / offsets from local variables defined in __remill_basic_block. The ideal fix for this would be a kind of auto-generated file that would tell us register names and their byte offsets in the State structures, and then we would use this to create __remill_basic_blocks contents. Making the program that makes this file would be relative easy, we just haven't got around yet to switching to this model.

[adahsuzixin]

After using the api_improvements branch, I met the configure problem.(P.S. I used libraries-llvm1000-ubuntu18.04-amd64.tar.xz in centos-amd64.) The CMake Error show:

[suzixin@localhost scripts]$ ./build.sh --debug --dyninst-frontend --prefix /home/suzixin/code/remill-install
[+] Enabling a debug build of remill
[+] Will build dyninst frontend
[+] New install directory is /home/suzixin/code/remill-install
[-] Library version is libraries-llvm1000-ubuntu1804-amd64
-- Install prefix: /home/suzixin/code/remill-install
CMake Error at cmake/BCCompiler.cmake:80 (message):
  The following compiler is not suitable to generate bitcode:
  /home/suzixin/initCode/remill/scripts/remill-build/libraries/llvm/bin/clang++
Call Stack (most recent call first):
  CMakeLists.txt:25 (include)

-- ccache: enabled with '/usr/local/bin/ccache'. The cache folder is located here: '/home/suzixin/.ccache'
 > Generating runtime target: x86
 > Generating runtime target: x86_avx
 > Generating runtime target: x86_avx512
 > Generating runtime target: amd64
 > Generating runtime target: amd64_avx
 > Generating runtime target: amd64_avx512
 > Generating runtime target: aarch64
-- Using LLVM_INSTALL_PREFIX hints for find_package(LLVM): /home/suzixin/initCode/remill/scripts/remill-build/libraries/llvm/lib/cmake/llvm/;/home/suzixin/initCode/remill/scripts/remill-build/libraries/llvm/share/llvm/cmake/
CMake Error at tools/CMakeLists.txt:29 (message):
  McSema depends upon Anvill being cloned into
  /home/suzixin/initCode/remill/tools/anvill

-- Configuring incomplete, errors occurred!
See also "/home/suzixin/initCode/remill/scripts/remill-build/CMakeFiles/CMakeOutput.log".
See also "/home/suzixin/initCode/remill/scripts/remill-build/CMakeFiles/CMakeError.log".

The CMakeError.log show:

/home/suzixin/initCode/remill/scripts/remill-build/libraries/llvm/bin/clang  -DCMAKE_HAVE_LIBC_PTHREAD    -rdynamic CMakeFiles/cmTC_87f51.dir/src.c.o  -o cmTC_87f51
CMakeFiles/cmTC_87f51.dir/src.c.o: In function `main':
src.c:(.text+0x35): undefined reference to `pthread_create'
src.c:(.text+0x41): undefined reference to `pthread_detach'
src.c:(.text+0x4d): undefined reference to `pthread_cancel'
src.c:(.text+0x5f): undefined reference to `pthread_join'
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[1]: *** [CMakeFiles/cmTC_87f51.dir/build.make:107: cmTC_87f51] Error 1
gmake[1]: Leaving directory '/home/suzixin/initCode/remill/scripts/remill-build/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:141: cmTC_87f51/fast] Error 2

To solve this problem, I tried to compile llvm and install in remill/scripts/remill-build/libraries, but it didn't help.

[adahsuzixin]

I haven't met this problem before. By the way, could you please add support for centos in cxx-common?

[pgoodman]

Ah! I think I forgot to mention that if you're using McSema, then you'll want the api_improvements branch of all three (Remill, Anvill, McSema) repositories. The next major release of McSema depends upon Anvill.

[pgoodman]

Instructions for cxx-common: https://gist.github.com/pgoodman/0882459dc2ecff0e674b79815e9301ff

[adahsuzixin]

if you're using McSema, then you'll want the api_improvements branch of all three (Remill, Anvill, McSema) repositories.

Hi, I tried to clone Anvill in remill/tools, unfortunately, met problem again when compiling.

Installing Anvill Python 3 API
cd /home/suzixin/trailofbits/remill/tools/anvill && which python3 && python3 setup.py install --force --user --prefix=
/usr/bin/python3
/home/suzixin/trailofbits/remill/tools/mcsema/mcsema/Arch/X86/Runtime/Runtime.cpp:31:10: fatal error: 'remill/Arch/X86/Runtime/State.h' file not found
#include "remill/Arch/X86/Runtime/State.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/mcsema_rt64-10.0.dir/build.make:88: recipe for target 'tools/mcsema/mcsema/Arch/X86/Runtime/runtime_64.o' failed
make[2]: *** [tools/mcsema/mcsema/Arch/X86/Runtime/runtime_64.o] Error 1
make[2]: Leaving directory '/home/suzixin/trailofbits/remill/scripts/remill-build'
CMakeFiles/Makefile2:1732: recipe for target 'tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/mcsema_rt64-10.0.dir/all' failed
make[1]: *** [tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/mcsema_rt64-10.0.dir/all] Error 2
running install
1 error generated.
tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/runtime_x86.dir/build.make:87: recipe for target 'tools/mcsema/mcsema/Arch/X86/Runtime/runtime_x86_Runtime.cpp.bc' failed
make[2]: *** [tools/mcsema/mcsema/Arch/X86/Runtime/runtime_x86_Runtime.cpp.bc] Error 1
make[2]: Leaving directory '/home/suzixin/trailofbits/remill/scripts/remill-build'
CMakeFiles/Makefile2:1786: recipe for target 'tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/runtime_x86.dir/all' failed
make[1]: *** [tools/mcsema/mcsema/Arch/X86/Runtime/CMakeFiles/runtime_x86.dir/all] Error 2

However, I can found remill/Arch/X86/Runtime/State.h in src directory.

[adahsuzixin]

This is the same question as issues/668 report. However, after trying api_improvements branch, another check failed.

[ 41%] Generating tests_x86_avx.bc
cd /home/suzixin/trailofbits/remill/scripts/remill-build/tests/X86 && ./lift-x86_avx-tests --arch x86_avx --bc_out tests_x86_avx.bc
F20200423 09:55:05.355089 20756 Arch.cpp:1127] Check failed: TotalOffset(dl, gep, state_ptr_type) <= reg.offset (2496 vs. 2488)
*** Check failure stack trace: ***
    @          0x1766bcc  google::LogMessageFatal::~LogMessageFatal()
    @           0xa29727  remill::Arch::CollectRegisters()
Aborted (core dumped)
tests/X86/CMakeFiles/run-x86_avx-tests.dir/build.make:88: recipe for target 'tests/X86/tests_x86_avx.bc' failed
make[3]: *** [tests/X86/tests_x86_avx.bc] Error 134
make[3]: Leaving directory '/home/suzixin/trailofbits/remill/scripts/remill-build'
CMakeFiles/Makefile2:2043: recipe for target 'tests/X86/CMakeFiles/run-x86_avx-tests.dir/all' failed
make[2]: *** [tests/X86/CMakeFiles/run-x86_avx-tests.dir/all] Error 2
make[2]: Leaving directory '/home/suzixin/trailofbits/remill/scripts/remill-build'
CMakeFiles/Makefile2:1252: recipe for target 'CMakeFiles/test_dependencies.dir/rule' failed
make[1]: *** [CMakeFiles/test_dependencies.dir/rule] Error 2
make[1]: Leaving directory '/home/suzixin/trailofbits/remill/scripts/remill-build'
Makefile:627: recipe for target 'test_dependencies' failed
make: *** [test_dependencies] Error 2