search

Hugal31 / yara-rust

Rust bindings for VirusTotal/Yara
Apache License 2.0
77 stars 29 forks source link

An error occurred when running the example compiled with the `musl` target #106

Closed bitcapybara closed 1 year ago

bitcapybara commented 1 year ago

Hi, It seems that after compiling to the musl target, an error occurs when running the example:

fish:

$ cargo run --target=x86_64-unknown-linux-musl --example tutorial
fish: Job 1, './target/x86_64-unknown-linux-m…' terminated by signal SIGSEGV (Address boundary error)

zsh:

$ cargo run --target=x86_64-unknown-linux-musl --example tutorial
zsh: segmentation fault (core dumped)  ./target/x86_64-unknown-linux-musl/release/examples/tutorial

But this problem does not occur when I compile directly using the cargo run --example tutorial command. I'm wondering if I'm overlooking some detail that's causing this problem, and how can I fix it?

environment

$ uname -a
Linux yanyuxing 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 GNU/Linux
ikrivosheev commented 1 year ago

@bitcapybara hello! Interesting... Does cargo clean help?

bitcapybara commented 1 year ago

en... cargo clean doesn't help on my computer. It seems that the yr_initialize function caused this problem, but I don't know why

Orycterope commented 1 year ago

By default, yara-rust will try to link against the yara lib installed on your system, as well as generate bindings from its .h. Maybe there is an ABI compatibility issue between the version of yara-rust you're using and the version of the yara lib installed on your system ?

Does 

YARA_CRYPTO_LIB=disable cargo run --target=x86_64-unknown-linux-musl --example tutorial --features yara-sys/vendored

still segfault ? Instead of linking against whichever version of the lib is present on your system, it will compile and link yaralib v4.2.3 from source (see Readme.md). This should rule out possible ABI compatibility issues with other versions.

Also, what version of the yara lib are you using ? You're in WSL2, but it doesn't tell us which distro you are using, and which version of libyara.

bitcapybara commented 1 year ago

I use ArchLinux and the version of yara is also up to date:

OS: Arch Linux on Windows 10 x86_64
Kernel: 5.15.90.1-microsoft-standard-WSL2
Uptime: 1 day, 15 hours, 21 mins
➤ sudo pacman -Si yara
Repository      : community
Name            : yara
Version         : 4.2.3-2
Description     : Tool aimed at helping malware researchers to identify and classify malware samples
Architecture    : x86_64
URL             : https://github.com/VirusTotal/yara
Licenses        : BSD
Groups          : None
Provides        : libyara.so=9-64
Depends On      : openssl  file  libmagic.so=1-64
Optional Deps   : None
Conflicts With  : None
Replaces        : None
Download Size   : 248.44 KiB
Installed Size  : 817.59 KiB
Packager        : Felix Yan <felixonmars@archlinux.org>
Build Date      : Wed 02 Nov 2022 07:06:06 AM CST
Validated By    : MD5 Sum  SHA-256 Sum  Signature

image

image

When I use the above command, another error occurs:

➤ YARA_CRYPTO_LIB=disable cargo run --target=x86_64-unknown-linux-musl --example tutorial --features yara-sys/vendored
   Compiling memchr v2.5.0
   Compiling regex-syntax v0.6.29
   Compiling same-file v1.0.6
   Compiling once_cell v1.17.1
   Compiling fnv v1.0.7
   Compiling cc v1.0.79
   Compiling fs_extra v1.3.0
   Compiling walkdir v2.3.3
   Compiling thread_local v1.1.7
   Compiling aho-corasick v0.7.20
   Compiling bstr v1.4.0
   Compiling nom v7.1.3
   Compiling regex v1.7.2
   Compiling cexpr v0.6.0
   Compiling globset v0.4.10
   Compiling bindgen v0.64.0
   Compiling ignore v0.4.20
   Compiling globwalk v0.8.1
   Compiling yara-sys v0.17.0 (/home/yanyuxing/github/yara-rust/yara-sys)
The following warnings were emitted during compilation:

warning: cc1: fatal error: /home/xxx/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/proc/linux.c: No such file or directory
warning: compilation terminated.

error: failed to run custom build command for `yara-sys v0.17.0 (/home/yanyuxing/github/yara-rust/yara-sys)`

Caused by:
  process didn't exit successfully: `/home/xxx/github/yara-rust/target/debug/build/yara-sys-2e2f5460b9b0dd8f/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("x86_64-unknown-linux-musl")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
  CC_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
  CC_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  RUSTC_LINKER = None
  cargo:rerun-if-env-changed=CROSS_COMPILE
  CROSS_COMPILE = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
  CFLAGS_x86_64-unknown-linux-musl = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
  CFLAGS_x86_64_unknown_linux_musl = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/include" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/modules" "-Wall" "-Wextra" "-Wno-deprecated-declarations" "-Wno-unused-parameter" "-Wno-unused-function" "-Wno-cast-function-type" "-Wno-type-limits" "-Wno-tautological-constant-out-of-range-compare" "-Wno-sign-compare" "-DUSE_LINUX_PROC=" "-DPOSIX=" "-DDOTNET_MODULE=1" "-DDEX_MODULE=1" "-DMACHO_MODULE=1" "-DNDEBUG=1" "-DYR_DEBUG_VERBOSITY=0" "-o" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/6e8863e52dceaa20-linux.o" "-c" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/proc/linux.c"
  cargo:warning=cc1: fatal error: /home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/proc/linux.c: No such file or directory
  cargo:warning=compilation terminated.
  exit status: 1

  --- stderr

  error occurred: Command "musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/include" "-I" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/modules" "-Wall" "-Wextra" "-Wno-deprecated-declarations" "-Wno-unused-parameter" "-Wno-unused-function" "-Wno-cast-function-type" "-Wno-type-limits" "-Wno-tautological-constant-out-of-range-compare" "-Wno-sign-compare" "-DUSE_LINUX_PROC=" "-DPOSIX=" "-DDOTNET_MODULE=1" "-DDEX_MODULE=1" "-DMACHO_MODULE=1" "-DNDEBUG=1" "-DYR_DEBUG_VERBOSITY=0" "-o" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/6e8863e52dceaa20-linux.o" "-c" "/home/yanyuxing/github/yara-rust/target/x86_64-unknown-linux-musl/debug/build/yara-sys-76cb8d58f8eabc26/out/yara/libyara/proc/linux.c" with args "musl-gcc" did not execute successfully (status code exit status: 1).
Orycterope commented 1 year ago

I do reproduce the segfault in my wsl2 ubuntu when running 

cargo run --target=x86_64-unknown-linux-musl --example tutorial

The core dump displays the following backtrace:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f83558fe58e in yara::internals::initialize () at src/internals/mod.rs:43
#2  0x00007f83558f7b09 in yara::initialize::InitializationToken::new () at src/initialize.rs:16
#3  0x00007f83558f39fb in yara::compiler::Compiler::new () at src/compiler.rs:43
#4  0x00007f83558f077a in tutorial::main () at examples/tutorial.rs:13

The segfault happens when trying to call yr_initialize():

mov rax, QWORD PTR [rip+0x6eec4]
call rax ; SEGFAULT

(gdb) p $rax
$3 = 0
(gdb) p $rip+0x6eec4
$4 = (*mut fn ()) 0x7f835596d452
(gdb) info files
...
        0x00007f835596d270 - 0x00007f835596e000 is .got
...

This looks like a linking error. I don't think libyara.so was loaded.

The crash does not happen when targeting x86_64-unknown-linux-gnu. I think this has to do with musl, .so files, and wsl2.

Orycterope commented 1 year ago

Ok I get it now. This has nothing to do with WSL. I can reproduce this on an ubuntu 22.04.

As I said, the default features try to link against the libyara.so which is installed on your system, but musl doesn't support this. Beside, this libyara.so is linked against the glibc (see ldd), so it won't work.

See this rusltang issue for more on this subject. The problem is exactly the same here: no interpreter is specified in .interp, which means the libyara.so is never loaded (see strace), and the program segfaults when trying to access the .got

If you target x86_64-unknown-linux-musl, you have to static link against libyara, you can't dynamic link, which is what the default features tries to do.

Either:

Use the vendored feature

As I previously said, the "vendored" feature will compile libyara and statically link against it. The error you have is because the submodule wasn't clone by git, you have to initialize it (we should really detect this in the build.rs)

git submodule init
git submodule update
YARA_CRYPTO_LIB=disable cargo run --target=x86_64-unknown-linux-musl --example tutorial --features yara-sys/vendored

Compile libyara.a by yourself

You can also compile libyara.a by yourself, and then statically link against it. You need to specify --features yara-static and set YARA_LIBRARY_PATH and YARA_INCLUDE_DIR to the path of the pre-compiled libyara.a and its header files.

See readme.md

In the build.rs, we should definitely detect the special case of musl target + dynamic linking, and abort the compilation.

ikrivosheev commented 1 year ago

@Orycterope maybe it's worth adding a check to yara-sys?

bitcapybara commented 1 year ago

@Orycterope Thanks for your explanation, this really helps me a lot!

I successfully ran the example with the command you provided:

git submodule init
git submodule update
cargo clean
YARA_CRYPTO_LIB=disable cargo run --target=x86_64-unknown-linux-musl --example tutorial --features vendored

On my computer, if I use the vendored feature, I must specify the YARA_CRYPTO_LIB=disable environment variable, otherwise, the compilation will still report an error.

Just a small suggestion, I think it would be better to mention this situation in README😊

Orycterope commented 1 year ago

Created 2 PRs:

When these two PRs will be merged, I think we can close this issue