Open g4ze opened 5 days ago
The analyzer works fine when i configure it to support all files. when i configure it to only support binaries and exe , it says incompatible mimetype. Moreover, the FE shows that the file type is application/x-executable
, which doesn't seem to be supported anywhere in models.
@mlodic any suggestions on what files should the analyzer support?
can you share the file that you used as a test?
how did you get the value application/x-executable
? cause that's deprecated and if you use IntelOwl it should classify it as application/vnd.microsoft.portable-executable
. On the contrary, if you run it locally, you could have an engine that tells you application/x-executable
which is the old one
ok I tried that tool and I think you should use application/vnd.microsoft.portable-executable
for that. Can you try with that and lmk?
I used application/vnd.microsoft.portable-executable only initially but frontend said incompatible mimetype. Does it work done in your local with accepted mime type as application/vnd.microsoft.portable-executable? If yes then I'll push a commit with that setting only.
here's a recording of what is exactly happening, in the first run as it can be seen, analyzer is set to accept all files. In the second run it is set to exe
only. Moreover, after the first run, file type is shown as application/x-executable
.
2024-06-28 21-18-10.zip
I think you found a bug. When we did this change, we wanted to use the correct mimetype value cause the x-
based one are not definitive and the new python library from python 3.11 started to use application/vnd.microsoft.portable-executable
instead of application/x-dosexec
. But it seems it is not the same for ELF executables and we need to revert this as soon as possible for ELF files, otherwise they are not supported.
Would you like to try to help us in solving this too?
If you check the old migrations, you can find all the analyzers where the x-executable
has been removed. We should re-add it.
Plus, we should re-add it as supported mimetype (there are 2 sections in the code where this is done, 1 backend and 1 frontend).
Then, you can use this mimetype for this analyzer too.
sure, i'll take a look. I did have a look at it before implementing the analyzer, when i was thinking for a way to check if the file was go-compiled as soon as we upload. It seems like the tool we are using is from magic import from_buffer as magic_from_buffer
and it stills gives application/x-executable
as an output for the go binary.
Here's a code snippet that serves the same implementation and lets us have an insight on how the tool reacts:
from magic import from_buffer as magic_from_buffer
files = ['/home/nilay/Documents/GitHub/byoc/main', "/home/nilay/Downloads/file.exe", "/home/nilay/Downloads/ping.elf"]
for file in files:
try:
with open(file, 'rb') as f:
file_type = magic_from_buffer(f.read(2048), mime=True)
print(f"{file}: {file_type}")
except IOError:
print(f"{file}: File not found or unreadable")
o/p:
/home/nilay/Documents/GitHub/byoc/main: application/x-executable
/home/nilay/Downloads/file.exe: application/vnd.microsoft.portable-executable
/home/nilay/Downloads/ping.elf: application/x-sharedlib
We have made implementations for SHARED_LIB and EXE but not anything like a go-compiled binary. Or anything which stills fals in the category of x-executable
Plus, we should re-add it as supported mimetype (there are 2 sections in the code where this is done, 1 backend and 1 frontend).
you mean somethings like the recent commit i made.
This works as expected on my machine.
x-executable
was being filtered and removed This works as expected on my machine.
i just realised that this fix is very naive and wont help the current users of intelowl who have already migrated their databases in the recent updates.
in the migration where we have removed x-executable, we have replaced two mimetypes with a single one.
application/x-dosexec
and application/x-executable
are replaced with newer application/vnd.microsoft.portable-executable
now the problem is that we want to bring back application/x-executable
as a supported type and add it to the old analyzers which supported it. The choke point is that we need to decide which all were the analyzers which supported application/x-executable
as one of their mimetypes. Currently, if an analyzer supports application/vnd.microsoft.portable-executable
, it is hard to decipher what exact mimetype it used to support previously in our two possible choices.
In my understanding, it leaves us two options:
application/vnd.microsoft.portable-executable
and blindly add application/x-executable
as one of the supported mimetype.
ORapplication/x-executable
in that time, use the resultant list to migrate and update all the specific analyzers.Do you have any suggestions or add-ons for this? Please correct me if I'm wrong with how I understood the problem.
I implemented the second option for the time being, here's what the quries returned:
intel_owl_db=# SELECT name
intel_owl_db-# FROM analyzers_manager_analyzerconfig
intel_owl_db-# WHERE 'application/x-executable' = ANY(supported_filetypes);
name
---------------
Capa_Info
ELF_Info
Malpedia_Scan
Intezer_Scan
Qiling_Linux
(5 rows)
intel_owl_db=# SELECT name
FROM analyzers_manager_analyzerconfig
WHERE 'application/x-executable' = ANY(not_supported_filetypes);
name
------
(0 rows)
Migrated the database based on the above o/p
yep you got everything right. and yes, those are the analyzers that were using that mimetype. You could also have found them just by looking for the string application/x-executable
in the old migrations
the test fails as application/vnd.microsoft.portable-executable
isn't necessarily compatible with the analyzer and our test file is surely not.
ah ok so I think we should modify the analyzer to avoid that the error that you mentioned {'error': 'Failed to parse file: failed to read pclntab: failed to locate pclntab'}
does make the analyzer fail. The result would be "success" and the report would be "this is not a Go-compiled file. Full error...". And we could add a warning log about that.
For all of the other errors triggered by the tool, it would fail the analyzer normally. In that way, the analyzer should never fail for any portable executables. Actually there's no better way to handle this and to avoid that this analyzer fails everytime a PE is sent to the platform, which is very common use case
closes #1451
Description
Please include a summary of the change and link to the related issue.
Type of change
Please delete options that are not relevant.
Checklist
develop
dumpplugin
command and added it in the project as a data migration. ("How to share a plugin with the community")test_files.zip
and you added the default tests for that mimetype in test_classes.py.FREE_TO_USE_ANALYZERS
playbook by following this guide.url
that contains this information. This is required for Health Checks._monkeypatch()
was used in its class to apply the necessary decorators.MockUpResponse
of the_monkeypatch()
method. This serves us to provide a valid sample for testing.Black
,Flake
,Isort
) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.tests
folder). All the tests (new and old ones) gave 0 errors.DeepSource
,Django Doctors
or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.Important Rules