Building Yari(-sys) on macOS arm64

3c7 commented 11 months ago

Hey, I saw #14 and tried to compile the project on my setup, however, I fail quite at the start. I have just a very basic Rust knowledge, so any Rust error make me panic a bit. :)

After compiling Yara, I tried to compile yari-sys which leads to the following binding.rs related error message:

error[E0119]: conflicting implementations of trait `std::fmt::Debug` for type `YR_SCAN_CONTEXT`
    --> /Users/<path>/yari/target/debug/build/yari-sys-04910e37d3f78597/out/bindings.rs:4776:10
     |
4776 | #[derive(Debug, Copy, Clone)]
     |          ^^^^^ conflicting implementation for `YR_SCAN_CONTEXT`
     |
    ::: yari-sys/src/bindings.rs:26:1
     |
26   | impl Debug for YR_SCAN_CONTEXT {
     | ------------------------------ first implementation here
     |
     = note: this error originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0119`.
error: could not compile `yari-sys` (lib) due to previous error

I have no idea about bindgen and the code generating the bindings in build.rs, but can help debugging. I attached the resulting binding.rs.txt as it may helps finding the issue.

MatejKastak commented 10 months ago

Hi Nils, thanks for showing interest in helping with this issue.

It seems that bindgen on your system is able to derive Debug on YR_SCAN_CONTEXT. Action to resolve this error should be to just delete my manual implementation of Debug.

You can locally delete this impl block. https://github.com/avast/yari/blob/master/yari-sys/src/bindings.rs#L26-L35 and then try to continue with build.

3c7 commented 10 months ago

Great, that worked! As the build seems to work for Windows and Linux, I marked the impl block for there OS only and I was able to compile yari-sys and yari-cli:

$ file yari
yari: Mach-O 64-bit executable arm64
$ ./yari
>> time.now()
Integer(1704192393)

However, I'm running into another issue now, as I want to create the python bindings:

$ CFLAGS="-I/opt/homebrew/lib/" LDFLAGS="-L/opt/homebrew/lib" LIBRARY_PATH="/opt/homebrew/lib" cargo build
[...]
ld: Undefined symbols:
            _PyBaseObject_Type, referenced from:
                _$LT$pyo3..pycell..PyCellBase$LT$U$GT$$u20$as$u20$pyo3..pycell..PyCellLayout$LT$T$GT$$GT$::tp_dealloc::hc62e1fc0f8a99c22 in yari.3cfxzg1694zul3op.rcgu.o
[...]
error: could not compile `yari-py` (lib) due to previous error

MatejKastak commented 10 months ago

Very nice!

I am afraid bare cargo won't be able to build the wheels. You will need to try install maturin.

Even though we don't use it yet, you can take inspiration from step in Github Action.

Here is the procedure I use for Windows

And here are the steps for Linux.

3c7 commented 10 months ago

Needed to trick a bit with the library path, but I got it to work:

📦 Built wheel for CPython 3.12 to /\/yari/target/wheels/yari_py-0.1.6-cp312-cp312-macosx_11_0_arm64.whl

>>> import yari
>>> c = yari.Context()
>>> c.eval("time.now()")
1704194493

So, the next step would be finding out how to implement this as Github action?

MatejKastak commented 10 months ago

Excellent! You are right, the next step would be to try to create a GitHub Action that will be able to build portable wheels and publish them as artifacts.

Here we need to make sure that the wheels are truly portable and don't depend on something from your system (statically linked).

Then the next GitHub Action after that will be able to follow and release those artifacts to PyPI.

Your help is greatly appreciated :pray: . It would be nice to document the procedure you used to setup environment and build the wheels.

3c7 commented 10 months ago

As a rough guideline:

I added link_macos() to build.rs:

diff --git a/yari-sys/build.rs b/yari-sys/build.rs
index 82b58bd..fcbe8c4 100644
--- a/yari-sys/build.rs
+++ b/yari-sys/build.rs
@@ -43,6 +43,15 @@ fn link_linux() {
     link_lib("z");
 }

+#[cfg(target_os = "macos")]
+fn link_macos() {
+    println!("cargo:rustc-link-lib=static=yara");
+    link_lib("crypto");
+    link_lib("magic");
+    link_lib("jansson");
+    link_lib("z");
+}
+
 fn main() {
     let ignored_macros = IgnoreMacros(
         vec![
@@ -60,6 +69,9 @@ fn main() {
     #[cfg(target_os = "linux")]
     link_linux();

+    #[cfg(target_os = "macos")]
+    link_macos();
+
     #[cfg(target_os = "windows")]
     link_windows();

As mentioned in https://github.com/avast/yari/issues/67#issuecomment-1873765339, I removed the Debug derivation for macOS - I don't know if this is necessary for x64 and why bindgen already has this?!

diff --git a/yari-sys/src/bindings.rs b/yari-sys/src/bindings.rs
index 7b5cf34..339ab70 100644
--- a/yari-sys/src/bindings.rs
+++ b/yari-sys/src/bindings.rs
@@ -23,6 +23,19 @@ pub struct SIZED_STRING {
     pub c_string: __IncompleteArrayField<::std::os::raw::c_char>,
 }

+#[cfg(target_os = "linux")]
+impl Debug for YR_SCAN_CONTEXT {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::result::Result<(), std::fmt::Error> {
+        // Basic implementation just to satisfy derive Debug
+        // We don't need all of the fields for now
+        f.debug_struct("YR_SCAN_CONTEXT")
+            .field("file_size", &self.file_size)
+            .field("entry_point", &self.entry_point)
+            .finish()
+    }
+}
+
+#[cfg(target_os = "windows")]
 impl Debug for YR_SCAN_CONTEXT {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::result::Result<(), std::fmt::Error> {
         // Basic implementation just to satisfy derive Debug

Then I was able to build everythin correctly using the following steps:

#!/usr/bin/env bash
set -e
# Envs
export CFLAGS="-I/opt/homebrew/include $CFLAGS"
export LDFLAGS="-L/opt/homebrew/lib $LDFLAGS"
export LIBRARY_PATH="/opt/homebrew/lib:$LIBRARY_PATH"

pushd yari-sys/yara
./bootstrap.sh
CFLAGS="-fPIC $CFLAGS" ./configure --enable-debug --disable-shared --enable-static --enable-magic --enable-dotnet --with-crypto
make clean && make
popd

pushd yari-sys
cargo build
popd

pushd yari-cli
cargo build
popd

pushd yari-py
maturin build -i python3.12
popd

However, a single Rust test fails:

warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions
       Fresh cfg-if v1.0.0
       Fresh bitflags v1.3.2
       Fresh unicode-ident v1.0.8
       Fresh once_cell v1.14.0
       Fresh libc v0.2.140
       Fresh lazy_static v1.4.0
       Fresh glob v0.3.0
       Fresh memchr v2.5.0
       Fresh proc-macro2 v1.0.53
       Fresh errno v0.2.8
       Fresh regex-syntax v0.6.29
       Fresh quote v1.0.26
       Fresh aho-corasick v0.7.20
       Fresh log v0.4.17
       Fresh minimal-lexical v0.2.1
       Fresh libloading v0.7.3
       Fresh syn v1.0.99
       Fresh nom v7.1.3
       Fresh regex v1.7.3
       Fresh either v1.8.0
       Fresh io-lifetimes v1.0.5
       Fresh cexpr v0.6.0
       Fresh which v4.3.0
       Fresh clang-sys v1.3.3
       Fresh lazycell v1.3.0
       Fresh shlex v1.1.0
       Fresh peeking_take_while v0.1.2
       Fresh rustc-hash v1.1.0
       Fresh target-lexicon v0.12.4
       Fresh rustix v0.36.8
       Fresh syn v2.0.8
       Fresh termcolor v1.1.3
       Fresh autocfg v1.1.0
       Fresh smallvec v1.10.0
       Fresh bindgen v0.64.0
       Fresh thiserror-impl v1.0.40
       Fresh is-terminal v0.4.3
       Fresh humantime v2.1.0
       Fresh tracing-core v0.1.29
       Fresh thiserror v1.0.40
       Fresh env_logger v0.10.0
       Fresh sharded-slab v0.1.4
       Fresh thread_local v1.1.4
       Fresh pyo3-build-config v0.18.2
       Fresh cc v1.0.79
       Fresh pin-project-lite v0.2.9
       Fresh io-lifetimes v0.7.3
       Fresh tracing-subscriber v0.3.15
       Fresh tracing v0.1.36
       Fresh adler v1.0.2
       Fresh gimli v0.26.2
       Fresh scopeguard v1.1.0
       Fresh tracing-error v0.2.0
       Fresh parking_lot_core v0.9.3
       Fresh lock_api v0.4.8
       Fresh addr2line v0.17.0
       Fresh miniz_oxide v0.5.4
       Fresh rustix v0.35.9
       Fresh nibble_vec v0.1.0
       Fresh pyo3-macros-backend v0.18.2
       Fresh object v0.29.0
       Fresh dirs-sys-next v0.1.2
       Fresh static_assertions v1.1.0
       Fresh strsim v0.10.0
       Fresh owo-colors v3.5.0
       Fresh indenter v0.3.3
       Fresh endian-type v0.1.2
       Fresh rustc-demangle v0.1.21
       Fresh clap_lex v0.4.1
       Fresh dirs-next v2.0.0
       Fresh radix_trie v0.2.1
       Fresh color-spantrace v0.2.0
       Fresh pyo3-macros v0.18.2
       Fresh eyre v0.6.8
       Fresh nix v0.26.2
       Fresh clap_builder v4.1.14
       Fresh backtrace v0.3.66
       Fresh parking_lot v0.12.1
       Fresh memoffset v0.8.0
       Fresh fd-lock v3.0.6
       Fresh pyo3-ffi v0.18.2
       Fresh unicode-width v0.1.10
       Fresh unindent v0.1.10
       Fresh indoc v1.0.7
       Fresh utf8parse v0.2.0
       Fresh unicode-segmentation v1.10.1
       Fresh color-eyre v0.6.2
       Fresh clap v4.1.14
       Fresh pyo3 v0.18.2
       Fresh rustyline v11.0.0
       Fresh yari-sys v0.1.6 (/<path>/yari/yari-sys)
       Fresh yari-cli v0.1.6 (/<path>/yari/yari-cli)
       Fresh yari-py v0.1.6 (/<path>/yari/yari-py)
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
     Running `/<path>/yari/target/debug/deps/yari-9880728f11ca49be`

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running `/<path>/yari/target/debug/deps/yari-708ca83f96c8d277`

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running `/<path>/yari/target/debug/deps/yari_sys-873156a85f2f2e1d`

running 157 tests
test bindings::bindgen_test_layout_RE ... ok
test bindings::bindgen_test_layout_RE_AST ... ok
test bindings::bindgen_test_layout_RE_CLASS ... ok
test bindings::bindgen_test_layout_RE_ERROR ... ok
test bindings::bindgen_test_layout_RE_FIBER ... ok
test bindings::bindgen_test_layout_RE_FAST_EXEC_POSITION ... ok
test bindings::bindgen_test_layout_RE_FAST_EXEC_POSITION_POOL ... ok
test bindings::bindgen_test_layout_RE_FIBER_LIST ... ok
test bindings::bindgen_test_layout_RE_FIBER_POOL ... ok
test bindings::bindgen_test_layout_RE_NODE ... ok
test bindings::bindgen_test_layout_RE_NODE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_RE_NODE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_AC_AUTOMATON ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH_LIST_ENTRY ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_4 ... ok
test bindings::bindgen_test_layout_YR_AC_STATE ... ok
test bindings::bindgen_test_layout_YR_ARENA ... ok
test bindings::bindgen_test_layout_YR_ARENA_BUFFER ... ok
test bindings::bindgen_test_layout_YR_ARENA_REF ... ok
test bindings::bindgen_test_layout_YR_ARRAY_ITEMS ... ok
test bindings::bindgen_test_layout_YR_ARRAY_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ATOM ... ok
test bindings::bindgen_test_layout_YR_ATOMS_CONFIG ... ok
test bindings::bindgen_test_layout_YR_ATOM_LIST_ITEM ... ok
test bindings::bindgen_test_layout_YR_ATOM_QUALITY_TABLE_ENTRY ... ok
test bindings::bindgen_test_layout_YR_ATOM_TREE ... ok
test bindings::bindgen_test_layout_YR_ATOM_TREE_NODE ... ok
test bindings::bindgen_test_layout_YR_DICTIONARY_ITEMS ... ok
test bindings::bindgen_test_layout_YR_DICTIONARY_ITEMS__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_DICT_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_INT_ENUM_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_INT_RANGE_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ITERATOR__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_MATCH ... ok
test bindings::bindgen_test_layout_YR_MATCHES ... ok
test bindings::bindgen_test_layout_YR_MEMORY_BLOCK ... ok
test bindings::bindgen_test_layout_YR_MEMORY_BLOCK_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_META ... ok
test bindings::bindgen_test_layout_YR_META__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_META__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_MODIFIER ... ok
test bindings::bindgen_test_layout_YR_MODULE ... ok
test bindings::bindgen_test_layout_YR_NAMESPACE ... ok
test bindings::bindgen_test_layout_YR_MODULE_IMPORT ... ok
test bindings::bindgen_test_layout_YR_NAMESPACE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_OBJECT ... ok
test bindings::bindgen_test_layout_YR_OBJECT_ARRAY ... ok
test bindings::bindgen_test_layout_YR_OBJECT_DICTIONARY ... ok
test bindings::bindgen_test_layout_YR_OBJECT_FUNCTION ... ok
test bindings::bindgen_test_layout_YR_OBJECT_FUNCTION__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_OBJECT_STRUCTURE ... ok
test bindings::bindgen_test_layout_YR_RELOC ... ok
test bindings::bindgen_test_layout_YR_PROFILING_INFO ... ok
test bindings::bindgen_test_layout_YR_RULE ... ok
test bindings::bindgen_test_layout_YR_RULES ... ok
test bindings::bindgen_test_layout_YR_RULES_STATS ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_RULE_PROFILING_INFO ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_4 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_5 ... ok
test bindings::bindgen_test_layout_YR_SCAN_CONTEXT ... ok
test bindings::bindgen_test_layout_YR_STRING ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_STRUCTURE_MEMBER ... ok
test bindings::bindgen_test_layout_YR_SUMMARY ... ok
test bindings::bindgen_test_layout_YR_VALUE ... ok
test bindings::bindgen_test_layout_YR_VALUE_STACK ... ok
test bindings::bindgen_test_layout__YR_COMPILER ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout__YR_FIXUP ... ok
test bindings::bindgen_test_layout__YR_HASH_TABLE ... ok
test bindings::bindgen_test_layout__YR_HASH_TABLE_ENTRY ... ok
test bindings::bindgen_test_layout__YR_LOOP_CONTEXT ... ok
test bindings::bindgen_test_layout__YR_MAPPED_FILE ... ok
test bindings::bindgen_test_layout__YR_STOPWATCH ... ok
test bindings::bindgen_test_layout__YR_STREAM ... ok
test bindings::bindgen_test_layout___darwin_pthread_handler_rec ... ok
test bindings::bindgen_test_layout___sFILE ... ok
test bindings::bindgen_test_layout___sbuf ... ok
test bindings::bindgen_test_layout__opaque_pthread_mutex_t ... ok
test bindings::bindgen_test_layout__opaque_pthread_t ... ok
test bindings::bindgen_test_layout_mach_timebase_info ... ok
test parser::tests::test_complex_expression_and ... ok
test parser::tests::test_complex_expression_without_context ... ok
test parser::tests::test_complex_expression_eq ... ok
test parser::tests::test_escape ... ok
test parser::tests::test_empty ... ok
test parser::tests::test_escape_char ... ok
test parser::tests::test_escape_dot ... ok
test parser::tests::test_escape_paren ... ok
test parser::tests::test_escape_plus ... ok
test parser::tests::test_escape_star ... ok
test parser::tests::test_function_call_args_empty_string ... ok
test parser::tests::test_function_call_args_ii ... ok
test parser::tests::test_function_call_args_one_regexp ... ok
test parser::tests::test_function_call_args_one_regexp_hard ... ok
test parser::tests::test_function_call_fff ... ok
test parser::tests::test_function_call_i ... ok
test parser::tests::test_function_call_identifier_with_numbers ... ok
test parser::tests::test_function_call_ignore_leading_whitespace ... ok
test parser::tests::test_function_call_ignore_trailing_whitespace ... ok
test parser::tests::test_function_call_ii ... ok
test parser::tests::test_function_call_no_args ... ok
test parser::tests::test_function_call_ri ... ok
test parser::tests::test_function_call_sf ... ok
test parser::tests::test_function_call_with_hexa_arguments ... ok
test parser::tests::test_identifier_array_access ... ok
test parser::tests::test_identifier_dictionary_access ... ok
test parser::tests::test_identifier_dot_before_iterable_access ... ok
test parser::tests::test_identifier_double_dot ... ok
test parser::tests::test_identifier_ends_with_dot ... ok
test parser::tests::test_identifier_starts_with_dot ... ok
test parser::tests::test_long_float ... ok
test parser::tests::test_parse_function_call_ri ... ok
test parser::tests::test_regexp ... ok
test parser::tests::test_regexp_escape2 ... ok
test parser::tests::test_regexp_escaped_forwardslash ... ok
test parser::tests::test_regexp_hard ... ok
test parser::tests::test_regexp_modifiers_combined ... ok
test parser::tests::test_regexp_modifiers_dot_all ... ok
test parser::tests::test_regexp_modifiers_invalid ... ok
test parser::tests::test_regexp_modifiers_no_case ... ok
test parser::tests::test_regexp_string ... ok
test parser::tests::test_regexp_no_modifiers ... ok
test parser::tests::test_rule_name ... ok
test parser::tests::test_string_count ... ok
test parser::tests::test_string_escaped_quotes ... ok
test parser::tests::test_string_match ... ok
test parser::tests::test_string_match_invalid_index ... ok
test parser::tests::test_string_match_invalid_index_use ... ok
test parser::tests::test_string_match_length ... ok
test parser::tests::test_string_match_missing_index ... ok
test parser::tests::test_string_match_offset ... ok
test parser::tests::test_string_missing_operation ... ok
test parser::tests::test_value ... ok
test parser::tests::test_value_with_context ... ok
test utils::tests::test_expression_simple ... ok
test utils::tests::test_expression_to_condition_simple_oneline ... ok
test utils::tests::test_expression_to_condition_oneline_with_string ... ok
test utils::tests::test_expression_to_condition_with_string ... ok

test result: ok. 157 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running `/<path>/yari/target/debug/deps/common-0b171babd2fab230`

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running `/<path>/yari/target/debug/deps/tests_dump_module-b7ad24865d79b8c2`

running 4 tests
error: test failed, to rerun pass `-p yari-sys --test tests_dump_module`

Caused by:
  process didn't exit successfully: `/<path>/yari/target/debug/deps/tests_dump_module-b7ad24865d79b8c2` (signal: 11, SIGSEGV: invalid memory reference)

The python tests look good:

============================= test session starts ==============================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /private/tmp/yari/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/nk/Documents/Development/yari/yari-py
collecting ... collected 22 items

tests/test_context.py::test_core PASSED                                  [  4%]
tests/test_context.py::test_raise[context_kwargs0] PASSED                [  9%]
tests/test_context.py::test_context_with_rule_path PASSED                [ 13%]
tests/test_eval.py::test_eval[hash.md5("dummy")-275876e34cf609db118f3d84b799a790] PASSED [ 18%]
tests/test_eval.py::test_eval_bool[time.now()] PASSED                    [ 22%]
tests/test_eval.py::test_eval_bool[hash.md5("dummy")] PASSED             [ 27%]
tests/test_eval.py::test_eval_pe_and_rule[pe.number_of_sections-4] PASSED [ 31%]
tests/test_eval.py::test_eval_pe_and_rule[hash.sha256(0, 100)-f852ce40ef76aae540d7e316271215235d984fef26359be2b25cfabea8da4ace] PASSED [ 36%]
tests/test_eval.py::test_eval_pe_and_rule[r|1 == 1-1] PASSED             [ 40%]
tests/test_eval.py::test_eval_pe_and_rule[r|$s00-1] PASSED               [ 45%]
tests/test_eval.py::test_eval_pe_and_rule[r|$s01-0] PASSED               [ 50%]
tests/test_eval.py::test_eval_pe_and_rule[r|#s00-1] PASSED               [ 54%]
tests/test_eval.py::test_eval_pe_and_rule[r|@s00[1]-1212] PASSED         [ 59%]
tests/test_eval.py::test_eval_pe_and_rule[r|!s00[1]-5] PASSED            [ 63%]
tests/test_eval.py::test_eval_rule_strings[r|$s00-True] PASSED           [ 68%]
tests/test_eval.py::test_eval_rule_strings[r|$s01-False] PASSED          [ 72%]
tests/test_eval.py::test_eval_raises[time.now] PASSED                    [ 77%]
tests/test_eval.py::test_eval_raises[invalid_module] PASSED              [ 81%]
tests/test_eval.py::test_eval_raises[time.not_now()] PASSED              [ 86%]
tests/test_eval.py::test_eval_raises[] PASSED                            [ 90%]
tests/test_eval.py::test_eval_with_missing_string_raises_exception PASSED [ 95%]
tests/test_eval.py::test_eval_with_missing_rule_raises_excpetion PASSED  [100%]

============================== 22 passed in 0.06s ==============================

I tried creating a naive Github Actions implementation in https://github.com/3c7/yari/blob/feature/macos_arm64/.github/workflows/test-macos.yml and then realized you're not able to build on Apple Silicon for free. 😬

Update: Also the second rust test with YARI_USE_BUNDLED_BINDINGS=1 cargo test --verbose fails because in this case the Debug derivation is needed. I'm confused haha

MatejKastak commented 10 months ago

then realized you're not able to build on Apple Silicon for free

That's a bummer :( I was not aware of that, maybe we have something enabled in https://github.com/avast organization. I will be able to try it tomorrow.

Also the second rust test with YARI_USE_BUNDLED_BINDINGS=1 cargo test --verbose fails because in this case the Debug derivation is needed. I'm confused haha

Yup there is one version of the bindings.rs that is manually generated. You can find it here. I usually generate them just by coping the bindings.rs that is generated during the compilation. In this case maybe we will need to have two different versions.

The reason for pre-generated bindings.rs is that not all systems where we use YARI have recent enough clang and LLVM (bindgen dependencies) to be able to generate bindings.

3c7 commented 10 months ago

But maybe another good news: example run for macos intel x64 where everything is working: https://github.com/3c7/yari/actions/runs/7388115313

done with this workflow:

name: Rust

on:
  push:
    branches: ["feature/macos_arm64"]

env:
  CARGO_TERM_COLOR: always

jobs:
  build:

    runs-on: macos-latest
    steps:
    - name: Checkout sources
      uses: actions/checkout@v3
      with:
        submodules: 'true'
    - name: Install Dependencies
      run: brew install autoconf automake libtool pkg-config jansson libmagic openssl python@3.10 python@3.11 python@3.12
    - name: Build Yara
      run: cd yari-sys/yara && ./bootstrap.sh && CFLAGS="-fPIC -I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" ./configure --enable-debug --disable-shared --enable-static --enable-cuckoo --enable-magic --enable-dotnet --with-crypto && make
    - name: Build Yari (yari-sys)
      run: cd yari-sys && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo build --verbose --release
    - name: Build Yari (yari-cli)
      run: cd yari-cli && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo build --verbose --release
    - name: Build Yari (yari-py)
      run: cd yari-py && python3.12 -mvenv venv-maturin && venv-maturin/bin/pip install maturin && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" venv-maturin/bin/maturin build -i python3.10 -i python3.11 -i python3.12
    - name: Run Rust tests
      run: CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo test --verbose

3c7 commented 10 months ago

Including all test from the original workflow now. In order to implement the test with bundled bindings, I added the following workaround and added a static binding file from another run:

diff --git a/yari-sys/build.rs b/yari-sys/build.rs
index fcbe8c4..c09c5f1 100644
--- a/yari-sys/build.rs
+++ b/yari-sys/build.rs
@@ -131,7 +131,12 @@ fn main() {
     let out_dir = env::var("OUT_DIR").unwrap();
     let out_path = PathBuf::from(out_dir).join("bindings.rs");
     if use_bundled_bindings.is_some() {
+        #[allow(unused_variables)]
         let binding_file = "bindings-unix.rs";
+
+        // Use different bindings on macOS.
+        #[cfg(target_os = "macos")]
+        let binding_file = "bindings-macos.rs";
         fs::copy(PathBuf::from("bindings").join(binding_file), out_path)
             .expect("Could not copy bindings to output directory");
     } else if let Some(bindings_file) = option_env!("YARI_BINDINGS_FILE") {

Job: https://github.com/3c7/yari/actions/runs/7388404344/job/20099169664 Yaml file: https://github.com/3c7/yari/blob/feature/macos_arm64/.github/workflows/test-macos.yml

3c7 commented 10 months ago

Played around a bit and hosted a macos arm runner myself. Compilation works, however, there are multiple issues with the tests:

warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions
    Finished test [unoptimized + debuginfo] target(s) in 0.03s
     Running unittests src/lib.rs (target/debug/deps/yari_sys-873156a85f2f2e1d)

running 157 tests
test bindings::bindgen_test_layout_RE ... ok
test bindings::bindgen_test_layout_RE_AST ... ok
test bindings::bindgen_test_layout_RE_CLASS ... ok
test bindings::bindgen_test_layout_RE_FAST_EXEC_POSITION_POOL ... ok
test bindings::bindgen_test_layout_RE_FAST_EXEC_POSITION ... ok
test bindings::bindgen_test_layout_RE_ERROR ... ok
test bindings::bindgen_test_layout_RE_FIBER ... ok
test bindings::bindgen_test_layout_RE_FIBER_LIST ... ok
test bindings::bindgen_test_layout_RE_FIBER_POOL ... ok
test bindings::bindgen_test_layout_RE_NODE ... ok
test bindings::bindgen_test_layout_RE_NODE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_RE_NODE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_AC_AUTOMATON ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH_LIST_ENTRY ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_AC_MATCH__bindgen_ty_4 ... ok
test bindings::bindgen_test_layout_YR_AC_STATE ... ok
test bindings::bindgen_test_layout_YR_ARENA ... ok
test bindings::bindgen_test_layout_YR_ARENA_BUFFER ... ok
test bindings::bindgen_test_layout_YR_ARENA_REF ... ok
test bindings::bindgen_test_layout_YR_ARRAY_ITEMS ... ok
test bindings::bindgen_test_layout_YR_ARRAY_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ATOM ... ok
test bindings::bindgen_test_layout_YR_ATOMS_CONFIG ... ok
test bindings::bindgen_test_layout_YR_ATOM_LIST_ITEM ... ok
test bindings::bindgen_test_layout_YR_ATOM_QUALITY_TABLE_ENTRY ... ok
test bindings::bindgen_test_layout_YR_ATOM_TREE ... ok
test bindings::bindgen_test_layout_YR_ATOM_TREE_NODE ... ok
test bindings::bindgen_test_layout_YR_DICTIONARY_ITEMS ... ok
test bindings::bindgen_test_layout_YR_DICTIONARY_ITEMS__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_DICT_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_EXTERNAL_VARIABLE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_INT_ENUM_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_INT_RANGE_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_ITERATOR__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_MATCH ... ok
test bindings::bindgen_test_layout_YR_MATCHES ... ok
test bindings::bindgen_test_layout_YR_MEMORY_BLOCK ... ok
test bindings::bindgen_test_layout_YR_MEMORY_BLOCK_ITERATOR ... ok
test bindings::bindgen_test_layout_YR_META ... ok
test bindings::bindgen_test_layout_YR_META__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_META__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_MODIFIER ... ok
test bindings::bindgen_test_layout_YR_MODULE ... ok
test bindings::bindgen_test_layout_YR_MODULE_IMPORT ... ok
test bindings::bindgen_test_layout_YR_NAMESPACE ... ok
test bindings::bindgen_test_layout_YR_NAMESPACE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_OBJECT ... ok
test bindings::bindgen_test_layout_YR_OBJECT_ARRAY ... ok
test bindings::bindgen_test_layout_YR_OBJECT_DICTIONARY ... ok
test bindings::bindgen_test_layout_YR_OBJECT_FUNCTION ... ok
test bindings::bindgen_test_layout_YR_OBJECT_FUNCTION__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_OBJECT_STRUCTURE ... ok
test bindings::bindgen_test_layout_YR_PROFILING_INFO ... ok
test bindings::bindgen_test_layout_YR_RELOC ... ok
test bindings::bindgen_test_layout_YR_RULE ... ok
test bindings::bindgen_test_layout_YR_RULES ... ok
test bindings::bindgen_test_layout_YR_RULES_STATS ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_RULES__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_RULE_PROFILING_INFO ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_4 ... ok
test bindings::bindgen_test_layout_YR_RULE__bindgen_ty_5 ... ok
test bindings::bindgen_test_layout_YR_STRING ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout_YR_SCAN_CONTEXT ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout_YR_STRING__bindgen_ty_3 ... ok
test bindings::bindgen_test_layout_YR_STRUCTURE_MEMBER ... ok
test bindings::bindgen_test_layout_YR_SUMMARY ... ok
test bindings::bindgen_test_layout_YR_VALUE ... ok
test bindings::bindgen_test_layout_YR_VALUE_STACK ... ok
test bindings::bindgen_test_layout__YR_COMPILER ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION__bindgen_ty_1 ... ok
test bindings::bindgen_test_layout__YR_EXPRESSION__bindgen_ty_2 ... ok
test bindings::bindgen_test_layout__YR_FIXUP ... ok
test bindings::bindgen_test_layout__YR_HASH_TABLE ... ok
test bindings::bindgen_test_layout__YR_HASH_TABLE_ENTRY ... ok
test bindings::bindgen_test_layout__YR_LOOP_CONTEXT ... ok
test bindings::bindgen_test_layout__YR_MAPPED_FILE ... ok
test bindings::bindgen_test_layout__YR_STOPWATCH ... ok
test bindings::bindgen_test_layout__YR_STREAM ... ok
test bindings::bindgen_test_layout___darwin_pthread_handler_rec ... ok
test bindings::bindgen_test_layout___sFILE ... ok
test bindings::bindgen_test_layout___sbuf ... ok
test bindings::bindgen_test_layout__opaque_pthread_mutex_t ... ok
test bindings::bindgen_test_layout__opaque_pthread_t ... ok
test bindings::bindgen_test_layout_mach_timebase_info ... ok
test parser::tests::test_complex_expression_and ... ok
test parser::tests::test_complex_expression_eq ... ok
test parser::tests::test_empty ... ok
test parser::tests::test_complex_expression_without_context ... ok
test parser::tests::test_escape ... ok
test parser::tests::test_escape_char ... ok
test parser::tests::test_escape_dot ... ok
test parser::tests::test_escape_paren ... ok
test parser::tests::test_escape_plus ... ok
test parser::tests::test_escape_star ... ok
test parser::tests::test_function_call_args_empty_string ... ok
test parser::tests::test_function_call_args_ii ... ok
test parser::tests::test_function_call_args_one_regexp ... ok
test parser::tests::test_function_call_args_one_regexp_hard ... ok
test parser::tests::test_function_call_i ... ok
test parser::tests::test_function_call_fff ... ok
test parser::tests::test_function_call_identifier_with_numbers ... ok
test parser::tests::test_function_call_ignore_leading_whitespace ... ok
test parser::tests::test_function_call_ignore_trailing_whitespace ... ok
test parser::tests::test_function_call_ii ... ok
test parser::tests::test_function_call_no_args ... ok
test parser::tests::test_function_call_sf ... ok
test parser::tests::test_function_call_ri ... ok
test parser::tests::test_function_call_with_hexa_arguments ... ok
test parser::tests::test_identifier_array_access ... ok
test parser::tests::test_identifier_dot_before_iterable_access ... ok
test parser::tests::test_identifier_double_dot ... ok
test parser::tests::test_identifier_dictionary_access ... ok
test parser::tests::test_identifier_ends_with_dot ... ok
test parser::tests::test_identifier_starts_with_dot ... ok
test parser::tests::test_long_float ... ok
test parser::tests::test_parse_function_call_ri ... ok
test parser::tests::test_regexp ... ok
test parser::tests::test_regexp_escape2 ... ok
test parser::tests::test_regexp_escaped_forwardslash ... ok
test parser::tests::test_regexp_hard ... ok
test parser::tests::test_regexp_modifiers_combined ... ok
test parser::tests::test_regexp_modifiers_dot_all ... ok
test parser::tests::test_regexp_modifiers_invalid ... ok
test parser::tests::test_regexp_modifiers_no_case ... ok
test parser::tests::test_regexp_no_modifiers ... ok
test parser::tests::test_regexp_string ... ok
test parser::tests::test_rule_name ... ok
test parser::tests::test_string_escaped_quotes ... ok
test parser::tests::test_string_count ... ok
test parser::tests::test_string_match ... ok
test parser::tests::test_string_match_invalid_index ... ok
test parser::tests::test_string_match_invalid_index_use ... ok
test parser::tests::test_string_match_length ... ok
test parser::tests::test_string_match_missing_index ... ok
test parser::tests::test_string_match_offset ... ok
test parser::tests::test_string_missing_operation ... ok
test parser::tests::test_value ... ok
test parser::tests::test_value_with_context ... ok
test utils::tests::test_expression_simple ... ok
test utils::tests::test_expression_to_condition_simple_oneline ... ok
test utils::tests::test_expression_to_condition_oneline_with_string ... ok
test utils::tests::test_expression_to_condition_with_string ... ok

test result: ok. 157 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running tests/common.rs (target/debug/deps/common-0b171babd2fab230)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests_dump_module.rs (target/debug/deps/tests_dump_module-b7ad24865d79b8c2)

running 4 tests
error: test failed, to rerun pass `-p yari-sys --test tests_dump_module`

Caused by:
  process didn't exit successfully: `/<path>/yari/target/debug/deps/tests_dump_module-b7ad24865d79b8c2` (signal: 11, SIGSEGV: invalid memory reference)
     Running tests/tests_eval.rs (target/debug/deps/tests_eval-d5035f2c759e3f3a)

running 38 tests
test test_create_context ... okerror: test failed, to rerun pass `-p yari-sys --test tests_eval`

Caused by:
  process didn't exit successfully: `/<path>/yari/target/debug/deps/tests_eval-d5035f2c759e3f3a` (signal: 11, SIGSEGV: invalid memory reference)
   Doc-tests yari-sys

running 7 tests
test yari-sys/src/parser.rs - parser::parse (line 355) ... ok
test yari-sys/src/yr_value.rs - yr_value::YrValue::is_undefined (line 37) ... ok
test yari-sys/src/lib.rs - Context::re_flags_from_modifier_string (line 808) ... ok
test yari-sys/src/parser.rs - parser::Expression<'_>::get_module (line 104) ... ok
test yari-sys/src/yr_value.rs - yr_value::bool::try_from (line 160) ... ok
test yari-sys/src/module.rs - module::Module::from_str (line 44) ... ok
test yari-sys/src/lib.rs - Context (line 436) ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.55s

error: 2 targets failed:
    `-p yari-sys --test tests_dump_module`
    `-p yari-sys --test tests_eval`

I assumed that only one test was failing, but with --no-fail-fast there are (at least) two tests failing.

MatejKastak commented 10 months ago

Tests

It seems that the majority of tests fail, we just don't see any traceback or anything. The most precise way to debug this issue would be to compile the tests and run them in gdb or rust-gdb. You can find the executables in the cargo test output on lines such as

[...]
     Running tests/tests_eval.rs (target/debug/deps/tests_eval-d5035f2c759e3f3a)
[...]

you will then be able to run the test and bt on SIGSEGV so see what we were doing.

There is also one easier option to try to get logs from tests. For this you need to run tests with RUST_LOG=debug and RUST_LOG=debug cargo test -- --nocapture --show-output --test-threads 1. You will also have to adjust the tests yari-sys/tests/common.rs

[...]
pub fn context() -> Context {
+   let _ = env_logger::builder().is_test(true).try_init();
    ContextBuilder::default().build().unwrap()
}
[...]

This is because we need to initialize logging system in tests (which is currently not done). Also the function context above in the example above, might not be called for every test. So if you still don't see the output you would have to sprinkle let _ = env_logger::builder().is_test(true).try_init(); on few more places. Best on the start of the tests where you want to see the logs.

Runners

Situation with the runners might be very tricky, I was not able to find if we have access to apple silicon runners or not.

One possible solution that comes to my mind is to cross-compile the wheel on x64 to arm. Based on my previous experience with this, I suspect it will be very painful.

If this was a pure Rust project than it would be much easier, but we would have to statically link and cross-compile the whole libyara.

3c7 commented 10 months ago

gdb seems not to be available for macOS arm64, I just used the easy way with LLDB and VS Codium. :)

Unfortunately yr_hash_table_lookup returns Null and therefore the call to self.modules.insert(k, v) crashes:

https://github.com/avast/yari/blob/b062de762af0b299621e26c0038c57b4a2825603/yari-sys/src/lib.rs#L743-L753

I need to recheck the Yara build, however, I'm pretty sure I included all modules there.

Update: Yeah, that happens for all Cuckoo module tests:

test test_cuckoo_dot_star_match ... FAILED
test test_cuckoo_regex_no_case ... FAILED
test test_cuckoo_no_match ... FAILED
test test_cuckoo_full_match ... FAILED

I probably messed up something.

3c7 commented 10 months ago

Yes, indeed I missed building the cuckoo module. Now all tests passing locally and they should also via Github Actions, I don't really know what happened there - (signal: 6, SIGABRT: process abort signal). My runner is still available and connected, don't know why the process aborted.

Regarding the Github hosted runners for macOS arm64: I also haven't found an overview of runner which you're able to use beside a general overview, however, there is only a single runner for macOS arm64 (macos-latest-xlarge) and it costs $0.16 per minute.

MatejKastak commented 10 months ago

Unfortunately yr_hash_table_lookup returns Null and therefore the call to self.modules.insert(k, v) crashes:

That is a very valuable find :+1: , I will try to improve the situation for the future and panic/return a proper error instead of crashing.

My runner is still available and connected, don't know why the process aborted.

The tests that are running are test_function_dump_cuckoo and test_function_dump_elf (judging by the fact that the other tests passed in the module). Might be some flaky test, I would not worry about it that much currently if it does not happen often.

Yes, indeed I missed building the cuckoo module.

Sorry, I should have spotted that and link you how to build libyara https://github.com/avast/yari/blob/master/yari-sys/README.md for YARI.

Regarding the Github hosted runners for macOS arm64: I also haven't found an overview of runner which you're able to use beside a general overview, however, there is only a single runner for macOS arm64 (macos-latest-xlarge) and it costs $0.16 per minute.

I will check, maybe we have some capacity here in the organization. The current blocker for the cross compilation was that even if we were to generate some python wheel on x64, we had no way of testing if it works on real arm silicon.

3c7 commented 10 months ago

The tests that are running are test_function_dump_cuckoo and test_function_dump_elf (judging by the fact that the other tests passed in the module). Might be some flaky test, I would not worry about it that much currently if it does not happen often.

They locally pass, so I assume the runner is a bit... unstable?

Sorry, I should have spotted that and link you how to build libyara https://github.com/avast/yari/blob/master/yari-sys/README.md for YARI.

Yeah, of course I built the lib but forgot about cuckoo. :)

I will check, maybe we have some capacity here in the organization. The current blocker for the cross compilation was that even if we were to generate some python wheel on x64, we had no way of testing if it works on real arm silicon.

Understood. If I can help shoot me a message. :)

3c7 commented 10 months ago

I've created an experimental brew formula so it's possible to build the yari binaries with brew install 3c7/tap/yari-macos (see the formula here). This, however, does not build and install the python libraries.

I still fight with ruby and the tests. :)

avast / yari

Building Yari(-sys) on macOS arm64 #67

Tests

Runners