steffenfritz / FileTrove

FileTrove indexes files and creates metadata from them.
https://filetrove.fritz.wtf
GNU Affero General Public License v3.0
37 stars 5 forks source link

[BUG] Yara-X-enabled ftrove dies with "Cannot allocate memory" in a relatively simple test case #98

Closed hkramski closed 1 month ago

hkramski commented 1 month ago

Describe the bug I built ftrove from an up-to-date repo against Yara-X 0.8.0 (as the current 0.9.0 seems to be too new atm) on Arch Linux and ran it over /usr/bin on my machine for testing (6122 files). At 51% it dies with

thread '<unnamed>' panicked at /home/kramski/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-19.0.1/src/runtime/func.rs:2349:14:
failed to create function: unable to make memory executable

Caused by:
    0: failed to make memory executable
    1: Cannot allocate memory (os error 12)

I'm using a relatively simple Yara rules file (I think):

rule Copyright
{
    strings:
        $a = "copyright"            nocase ascii wide
        $b = "(c)"                  nocase ascii wide
        $c = "all rights reserved"  nocase ascii wide

    condition:
        1 of them
}

rule CC0
{
    strings:
        $a = /cc[- ]?0/             nocase ascii wide
        $b = "public domain"        nocase ascii wide
        $c = "no rights reserved"   nocase ascii wide
        $d = "no copyright"         nocase ascii wide

    condition:
        1 of them
}

rule CC_BY
{
    strings:
        $a = /cc[- ]by/             nocase ascii wide

    condition:
        1 of them
}

rule CC_BY_ND
{
    strings:
        $a = /cc[- ]by[- ]nd/       nocase ascii wide

    condition:
        1 of them
}

rule CC_BY_NC
{
    strings:
        $a = /cc[- ]by[- ]nc/       nocase ascii wide

    condition:
        1 of them
}

rule CC_BY_NC_SA
{
    strings:
        $a = /cc[- ]by[- ]nc[- ]sa/ nocase ascii wide

    condition:
        1 of them
}

rule CC_BY_NC_ND
{
    strings:
        $a = /cc[- ]by[- ]nc[- ]nd/ nocase ascii wide

    condition:
        1 of them
}

To Reproduce Steps to reproduce the behavior:

  1. Build the Yara-X C-Library Version 0.8.0 from https://github.com/VirusTotal/yara-x/archive/refs/tags/v0.8.0.zip following the instructions at https://virustotal.github.io/yara-x/docs/api/c/c-/#building-the-c-library.
  2. Build ftrove following the instructions here
  3. Install ftrove following the instructions here
  4. Add yara/licenses.yara to the ftrove install folder, containing the rules above
  5. run ftrove -i /usr/bin -y yara/licences.yara

Expected behavior ftrove should complete without fatal errors

Additional context The machine has 48 GB RAM. filetrove.log only shows some complains about "permission denied" on a few files.

hkramski commented 1 month ago

Big Ooops: There still was an older ftrove in the path built against Yara-X 0.3.0 which indeed had this problem.

All is well using Yara-X 0.8.0.

Sorry for the noise.

hkramski commented 1 month ago

There still was an older ftrove in the path built against Yara-X 0.3.0 which indeed had this problem. All is well using Yara-X 0.8.0. Sorry for the noise.