search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
940 stars 218 forks source link

Parsers for phone artifacts integrating ALeapp/iLeapp #43

Open lfcnassif opened 4 years ago

lfcnassif commented 4 years ago

Currently we just have parsers for WhatsApp and Skype (edited: and Telegram). To decrease the dependency of other tools (UFDR reports), it is important to have parsers for calls, contacts, calendar, sms/mms, notes, locations, other instant message apps (facebook, telegram, instagram, twitter, snapchat...), custom email containers. Android and iOS will need different parsers. This ticket could be broken in smaller ones for each artifact.

Contributions are very welcome :)

lfcnassif commented 4 years ago

Some logic or code for different Android artifacts could be adapted from https://github.com/sleuthkit/autopsy/tree/develop/InternalPythonModules/android as their license is Apache v2.

Currently IPED has support for python tasks, parsers not yet, but I could add if anyone from python world is interested in contributing. But a number of those artifacts could be decoded in tasks instead of parsers, basically those that will create new case items corresponding to database records (calls, contacts, calendar, sms) storing relevant info in new metadata columns. Items with small text (like sms) could also store text as metadata. Items with very large text to be indexed should be decoded by a parser implementation, except if a subitem is created to store the text by the task.

hauck-jvsh commented 4 years ago

I think that the Telegram/Videogram parser fits here.

lfcnassif commented 4 years ago

For sure! There is a specific ticket #177 to track the progress on the telegram parser. Thanks, @hauck-jvsh!

lfcnassif commented 3 years ago

Just found this iOS artifacts python parser MIT licensed https://github.com/abrignoni/iLEAPP

lfcnassif commented 3 years ago

https://github.com/kacos2000/Queries

lfcnassif commented 2 years ago

Just found this iOS artifacts python parser MIT licensed https://github.com/abrignoni/iLEAPP

And this for Android: https://github.com/abrignoni/ALEAPP

lfcnassif commented 2 years ago

MIT licensed: https://github.com/den4uk/andriller

lfcnassif commented 11 months ago

Just to warn other DEVs and avoid duplicate efforts, @patrickdalla is working on this and should share some ideas and his progress here soon for comments/suggestions.

patrickdalla commented 11 months ago

I pushed a branch where I'm working. Still draft not tested on windows and with initial Aleap integration

Em sex., 6 de out. de 2023 14:35, Luis Filipe Nassif < @.***> escreveu:

Just to warn other DEVs and avoid duplicate efforts, @patrickdalla https://github.com/patrickdalla is working on this and should share some ideas and his progress here soon for comments/suggestions.

— Reply to this email directly, view it on GitHub https://github.com/sepinf-inc/IPED/issues/43#issuecomment-1751235573, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG247SZICZVVOGWOJIACNFDX6BFN5AVCNFSM4LPITZ2KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZVGEZDGNJVG4ZQ . You are receiving this because you were mentioned.Message ID: @.***>

patrickdalla commented 11 months ago

I noted that, as I downloaded ALeapp scripts via git, it included git config files, and eclipse PUSH did not recognize these scripts as part of IPED, not pushing them. Currently (locally) I am embbeding them inside "scripts/tasks/ALEAPP" folder. As the ALeapp license is MIT, I think it can stay as it is. Any objection @lfcnassif @hauck-jvsh ?

Another important note about these scripts is that, to "override" the html generation with IPED items generation code, I had to overwrite the script "scripts/artifact_report.py" with a IPED java class wrapper. So, for any future ALeapp update, we must remember to overwrite this file again.

patrickdalla commented 11 months ago

I could find some code that hooks Python module loading, and redirect to a java code to make this change "on-the-fly". This could be and option to avoid this ALeapp upgrade procedure. But, although worked for many python modules, for the ALeapp modules there were some exceptions thrown for which I could not identify the cause/problem, yet. Do you think it worth trying to implement this option, @lfcnassif ?

lfcnassif commented 11 months ago

Currently (locally) I am embbeding them inside "scripts/tasks/ALEAPP" folder. As the ALeapp license is MIT, I think it can stay as it is. Any objection @lfcnassif @hauck-jvsh ?

No problem from my side. Another possible option would be to put aleapp into iped/tools folder and reference it from the iped task.

Do you think it worth trying to implement this option, @lfcnassif ?

If it is possible to simplify the dependency upgrade process, I think it is worth to try, instead of having to maintain a dependency patch/fork...

patrickdalla commented 11 months ago

I decided to put ALeapp Scripts in tools. They will be downloaded by maven build, avoiding code redudancy inside IPED project tree.

lfcnassif commented 11 months ago

I decided to put ALeapp Scripts in tools. They will be downloaded by maven build, avoiding code redudancy inside IPED project tree.

Nice!

patrickdalla commented 11 months ago

I could implement the Python module load hook. So, the changes needed in ALeapp code are made by IPED when loading them. It is working and implemented in class PythonHook. I tryed to implement PythonHook in a decoupled way from ALeapp, so it can be used by some other Python tools. Unfortunatelly JEP does not support the method call with keywords as parameters (named parameters) of java objects. So I had to make a specific method for ALeapp modification.

patrickdalla commented 11 months ago

Next steps: 1) Test on Windows machine 2) Map aleap html fields on IPED default fields. I will implement this to be configured on ALeappConfig.txt. Any objection? 3) Detail specific ALeapp plugins resources. For example, some plugins pass a link to a existing file as an HTML field of the artifact table. This link can be parsed and also saved as such in LINKED_ITEMS metadata.

patrickdalla commented 8 months ago

Hi @lfcnassif. How should we install ALeapp dependecies in IPED distribution? Could we install on demand, on ALeappTask initialization? Or can we already make them (maybe some) available in the python embedded distribution?

lfcnassif commented 8 months ago

Depending on the total size of the dependencies, I think we can include the small ones (and the ones difficult to install) by default. If total size is not that big, since this is a very important feature in my opinion, maybe we can include all of them.

lfcnassif commented 8 months ago

I would put big native dependencies apart (like face recognition and audio transcription) in the python package distributed separately, since they wouldn't work on Linux machines. I guess it is not the case here. Portable dependencies can be included by default, if they are not that large.

patrickdalla commented 8 months ago

There are two more processing steps that I think it would be better, for maintenance purpose, to place separately in a JavaScript task.

The LeappBridge task contains many code to execute python code. It can be confuse to the maintainer if we mix with the code to make these steps.

Any objection or better idea, @lfcnassif ?

lfcnassif commented 8 months ago

The first seems fine to me, since we already use a javascript task to classify artifacts based on some properties, the same task could be used to centralize things, maybe some conflict could happen with existing rules and it could make the resolution easier.

Not sure about the second. I would put it in a Java class, called by the main ALeap task. It may also give better performance depending on the number of Metadata values to map.

lfcnassif commented 8 months ago

Another approach to remap ALeapp extracted Metadata is to use the MetadataUtil class into iped-parsers package, since it already remaps many Metadata keys and values today.

patrickdalla commented 8 months ago

Consider the following hypothesis: Some advanced user, that knows better python that java, wants to make some ALeapp plugin. After the plugin is ready, he wants to make this categorization and metadata mapping. Or even the user wants to update ALeapp with new plugins, with new extracted items and corresponding metadatas. It would be easier to him if we allow this mappings in some easier script file, without the need of IPED team approval and recompilation.

In this scenario, a Python script task would be even better although I have some difficulty with python. The javascript file would be simpler for me.

But we could also design some configuration file. For categorization, maybe some enhancement on CategoriesConfig would be enough and useble for other modules also, like accepting some other metadata rules instead of only mimetype.