elf support - Githubissues

mr-tz commented 7 years ago

Test candidates: A47FD9031D7C45AB61F9E8C68458BA81, 1D081F79D698DBDE8FBD8B6C857ECCA8, and 164EF16A5257311859822765778383D0 (all from the same malware family).

mr-tz commented 2 years ago

here is a good example of an obfuscated elf binary: a0cd554c35dee3fed3d1607dc18debd1296faaee29b5bd77ff83ab6956a6f9d6 https://www.intezer.com/blog/research/new-linux-threat-symbiote/

https://github.com/mandiant/flare-floss/issues/276#issuecomment-1168244301

symbolicvoid commented 1 year ago

I was wondering about this issue. @mr-tz told me a little bit about this on an e-mail and pointed me to the capa PR that added ELF support, and it seems easy. Majority of the changes there seem to be related to functionality of capa, so is there something in FLOSS that works for PE but might not work for ELF files? Or could this be as simple as just allowing FLOSS to analyze ELF files?

williballenthin commented 1 year ago

i think enabling the support via vivisect is probably pretty easy, as you've found. the bigger concern is being able to prove its working, which involves finding enough test samples to demonstrate that FLOSS does something useful. if we can find 5-10 samples/techniques, then i'd be happy to merge this support.

mr-tz commented 1 year ago

The testfiles repo should contain (or we can easily add them) the test samples generated from source.

symbolicvoid commented 1 year ago

i think enabling the support via vivisect is probably pretty easy, as you've found. the bigger concern is being able to prove its working, which involves finding enough test samples to demonstrate that FLOSS does something useful. if we can find 5-10 samples/techniques, then i'd be happy to merge this support.

By that, do you mean demonstrating ELF files have obfuscated strings, or demonstrating that FLOSS can extract them from ELF files? Or do we just want more ELF malware samples to test for obfuscated strings to prove that ELF support is actually useful?

The testfiles repo should contain (or we can easily add them) the test samples generated from source.

Do you mean we could use the test samples in the testfiles repo for this as well?

mr-tz commented 1 year ago

If we can prove this works for the testfiles (e.g. https://github.com/mandiant/flare-floss-testfiles/tree/master/src/decode-base64/bin) I'd be pretty happy already.

williballenthin commented 10 months ago

@c-urly this is the issue that you're hoping to work on. please review this thread and propose a plan 😁

the other threads you commented on were specifically for (1) QUANTUMSTRAND, an experimental rendering mode, and (2) Go language string support.

this thread is about deobfuscating strings found in ELF files.

c-urly commented 10 months ago

Thanks, @williballenthin will go through it and will ask any questions i will have on this thread.

c-urly commented 10 months ago

@williballenthin So here is what I have understood so far, Please point me out if I am wrong somewhere.

We currently only support PE files as SUPPORTED_FILE_MAGIC only has PE header.
I think first I will have to check that by adding ELF in SUPPORTED_FILE _MAGIC macro, the code deobfuscates the elf malware or not, if there is any problem I will have to dig deeper into each string deobfuscation And I will have to compare floss performance with strings (let me know if there is any other way to check floss' performance).
I will also have to add code to load the elf file in the Vivisect tool, I am still checking this.

c-urly commented 10 months ago

Does this look fine? Curently I have made these changes and tested on some flare elf testfiles.

diff --git a/floss/const.py b/floss/const.py index 3369880..24140f7 100644 --- a/floss/const.py +++ b/floss/const.py @@ -3,7 +3,7 @@ KILOBYTE = 1024 MEGABYTE = 1024 KILOBYTE MAX_FILE_SIZE = 16 MEGABYTE -SUPPORTED_FILE_MAGIC = {b"MZ"} +SUPPORTED_FILE_MAGIC = {b"MZ",b"\x7fELF"} MIN_STRING_LENGTH = 4 MAX_STRING_LENGTH = 2048

diff --git a/floss/main.py b/floss/main.py index a49edea..5157ff3 100644 --- a/floss/main.py +++ b/floss/main.py @@ -356,10 +356,12 @@ def is_supported_file_type(sample_file_path: Path): :return: True if file type is supported, False otherwise """ with sample_file_path.open("rb") as f:

magic = f.read(2)
magic = f.read(4)
print(magic[:2]) if magic in SUPPORTED_FILE_MAGIC: return True
elif magic[:2] in SUPPORTED_FILE_MAGIC:
return True else: return False

$ floss test-decode-split-stackstrings INFO: floss: extracting static strings .. analyzing program finding decoding function features: 100%|██████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 488.13 functions/s, skipped 0 library functions] INFO: floss.stackstrings: extracting stackstrings from 11 functions INFO: floss.results: goodbye INFO: floss.results: goodbye world INFO: floss.results: goodbye moon extracting stackstrings: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 97.40 functions/s] INFO: floss.tightstrings: extracting tightstrings from 0 functions... extracting tightstrings: 0 functions [00:00, ? functions/s] INFO: floss.string_decoder: decoding strings emulating function 0x400418 (call 1/1): 100%|████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:01<00:00, 10.08 functions/s] INFO: floss: finished execution after 14.08 seconds INFO: floss: rendering results

FLARE FLOSS RESULTS (version 3.0.1)

─────────────────────────── FLOSS STATIC STRINGS (79) ───────────────────────────

+----------------------------------+ | FLOSS STATIC STRINGS: ASCII (78) | +----------------------------------+

/lib64/ld-linux-x86-64.so.2 libc.so.6 printf rand libc_start_main gmon_start GLIBC2.2.5 UH-H UH-H []A\A]A^A ;*3$" GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4) .symtab .strtab .shstrtab .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .text .fini .rodata .eh_frame_hdr .eh_frame .init_array .fini_array .jcr .dynamic .got .got.plt .data .bss .comment crtstuff.c __JCR_LIST deregister_tm_clones register_tm_clones do_global_dtors_aux completed.6973 __do_global_dtors_aux_fini_array_entry frame_dummy frame_dummy_init_array_entry test-decode-split-stackstrings.c FRAME_END JCR_END init_array_end _DYNAMIC init_array_start _GLOBAL_OFFSETTABLE libc_csu_fini _ITM_deregisterTMCloneTable data_start _edata _fini printf@@GLIBC_2.2.5 __libc_start_main@@GLIBC_2.2.5 data_start gmon_start dso_handle _IO_stdin_used libc_csu_init _end _start bss_start main _Jv_RegisterClasses TMC_END__ _ITM_registerTMCloneTable _init rand@@GLIBC_2.2.5

+------------------------------------+ | FLOSS STATIC STRINGS: UTF-16LE (1) | +------------------------------------+

@8\t@

───────────────────────── FLOSS STACK STRINGS (3) ─────────────────────────

goodbye goodbye world goodbye moon

───────────────────────── FLOSS TIGHT STRINGS (0) ─────────────────────────

─────────────────────────── FLOSS DECODED STRINGS (0) ───────────────────────────

c-urly commented 10 months ago

Hi @williballenthin , @mr-tz , How to run all the test cases in flare-floss-testfiles with pytest?

williballenthin commented 10 months ago

pytest tests/

you can also check out the CI config here for a more detailed walkthrough: https://github.com/mandiant/flare-floss/blob/master/.github/workflows/tests.yml

c-urly commented 10 months ago

I have not checked this on actual malware. I would appreciate any help from you all to download the given malware. All test files under flare-floss-test files/src/ passed for Linux and Windows.

mr-tz commented 10 months ago

If the test files pass that's a great start. Can you clarify what you mean for the actual malware samples (i.e. how to download the samples mentioned in https://github.com/mandiant/flare-floss/issues/40#issuecomment-286215095)?

c-urly commented 10 months ago

Yes, how to download sample mentioned in the comments.

mr-tz commented 10 months ago

Hm, it seems the ones I've mentioned are private-only samples. Let's focus on the test samples generated from source for now.

c-urly commented 10 months ago

@mr-tz Pl check the PR in the test repo, https://github.com/mandiant/flare-floss-testfiles/pull/17

mandiant / flare-floss

elf support #40