Proposing llvm-lit for testing

weliveindetail commented 1 year ago

This patch proposes a test infrastructure based on llvm-lit for O-MVLL. Requirements seem to be sufficiently close and lit provides many useful features like:

REQUIRES to detect host features or supported LLVM targets
XFAIL to indicate bugs or incomplete implementations
flexible RUN and CHECK lines that execute multiple tests on one test input
regex patterns and matchers

Documentation in upstream LLVM can be found here:

for the driver llvm-lit: https://llvm.org/docs/CommandGuide/lit.html
for the validator FileCheck: https://llvm.org/docs/CommandGuide/FileCheck.html

This approach is fairly standard for out-of-tree tools that have LLVM as their prevailing dependency. Please have a look at src/test/passes/arithmetic for examples and find more details and instructions in the commit message.

The PR requires https://github.com/open-obfuscator/o-mvll/pull/11 and should be rebased once that one landed. The duplicate commits should then disappear.

weliveindetail commented 1 year ago

From my side this would be ready to land. Let me know if you have any questions. I am happy to discuss every little detail. Eventually, it would be great to run the regression tests for each PR.

Test output on Ubuntu 22.04 with NDK-25c (building with stage-1 clang, testing with stage-2 clang):

$ ninja check
[0/1] Running O-MVLL regression tests
Running tests with: /path/to/llvm-toolchain-android/out_r25c/stage2/bin/clang
Testing plugin file: /path/to/o-mvll-build-ndk-r25c/libOMVLL.so
OMVLL_PYTHONPATH: /path/to/Python-3.10.7/Lib/
-- Testing: 4 tests, 4 workers --
PASS: O-MVLL Tests :: passes/arithmetic/xor-x86_64-darwin.c (1 of 4)
PASS: O-MVLL Tests :: passes/arithmetic/xor-x86_64-linux.c (2 of 4)
PASS: O-MVLL Tests :: passes/strings-encoding/basic-aarch64.cpp (3 of 4)
PASS: O-MVLL Tests :: passes/arithmetic/xor-aarch64.c (4 of 4)

Testing Time: 0.56s
  Passed: 4

Test output on macOS 13.3.1 with MacOSX SDK 13.0 (building with AppleClang, same test results for both below compilers):

➜ ninja check
[0/1] Running O-MVLL regression tests
Running tests with: /path/to/llvm-project/build-macosx13.0-release/bin/clang
Testing plugin file: /path/to/o-mvll-build-macosx13.0/libOMVLL.dylib
OMVLL_PYTHONPATH: /path/to/Python-3.10.7/Lib
-- Testing: 4 tests, 4 workers --
PASS: O-MVLL Tests :: passes/arithmetic/xor-x86_64-linux.c (1 of 4)
PASS: O-MVLL Tests :: passes/arithmetic/xor-x86_64-darwin.c (2 of 4)
XFAIL: O-MVLL Tests :: passes/strings-encoding/basic-aarch64.cpp (3 of 4)
PASS: O-MVLL Tests :: passes/arithmetic/xor-aarch64.c (4 of 4)
********************
Expectedly Failed Tests (1):
  O-MVLL Tests :: passes/strings-encoding/basic-aarch64.cpp

Testing Time: 1.16s
  Passed           : 3
  Expectedly Failed: 1

➜ /path/to/llvm-project/build-macosx13.0-release/bin/clang --version | head -1
clang version 14.0.0 (https://github.com/apple/llvm-project c41f13252ed4b49f246729b4d91ff521d5a6bf9d)

➜ /path/to/llvm-project/build-macosx13.0-release/bin/clang --version | head -1
Apple clang version 14.0.0 (clang-1400.0.29.202)

I haven't had the chance to test this on macOS ARM, but I don't expect major differences.

romainthomas commented 1 year ago

Thank you @weliveindetail, It looks good on my side. I also experienced the same issue with the string obfuscation pass on the Apple M1 but I think this issue should be addressed with some changes in the design of the JIT.

open-obfuscator / o-mvll

Proposing llvm-lit for testing #12