search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
924 stars 217 forks source link

Parser for Threema Secure Messenger files #1896

Closed costa-arc closed 9 months ago

costa-arc commented 11 months ago

Currently working for IOS filesystem (main database: ThreemaData.sqlite) Recovering deleted data from sqlite and tests for this parser not implemented yet

lfcnassif commented 11 months ago

Thank you @costa-arc for this Threema parser contribution!

lfcnassif commented 9 months ago

Hi @patrickdalla, I'm changing the reviewer of this to me and assigning #1956 to you when you finish the other ones, thanks!

@costa-arc, sorry for our delay here. I'll start a crawling for Threema database samples in our 500 cases database so we can test this properly.

lfcnassif commented 9 months ago

The crawling finished and after scanning ~800 cases, I just found Threema databases in 2 cases... @tc-wleite, when you have some time, could you search for Threema databases in São Paulo cases database? I used the search query below: name:(threema ThreemaData) && type:(plist sqlite)

wladimirleite commented 9 months ago

@tc-wleite, when you have some time, could you search for Threema databases in São Paulo cases database?

Sure! Just start a search here.

lfcnassif commented 9 months ago

I just reverted some unrelated changes and pushed.

Sure! Just start a search here.

Thanks @tc-wleite! When it finishes, you can send the samples to me by Teams, thanks!

wladimirleite commented 9 months ago

Thanks @tc-wleite! When it finishes, you can send the samples to me by Teams, thanks!

Unfortunately, I found Threema files in only 2 cases (out of 2,000). And by the length of the files, they don't seem to contain much information (if any). Anyway, I just sent them to you by Teams.

lfcnassif commented 9 months ago

Unfortunately, I found Threema files in only 2 cases (out of 2,000). And by the length of the files, they don't seem to contain much information (if any). Anyway, I just sent them to you by Teams.

Thank you! Those I got are bigger, one has 1.5MB and other 68MB.

lfcnassif commented 9 months ago

Hi @costa-arc, I'm reviewing this PR, it is well written and has most features of other chat parsers, thank you very much! I'm just taking care of a few details.

One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.

lfcnassif commented 9 months ago

One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.

I just saw it is possible to be outside the database if data field is 36 bytes size, which means it's the external file name. Let me know if I misunderstood.

costa-arc commented 9 months ago

One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.

I just saw it is possible to be outside the database if data field is 36 bytes size, which means it's the external file name. Let me know if I misunderstood.

Exactly @lfcnassif, larger files are stored outside the database and checking data field is the best way to find out.

lfcnassif commented 9 months ago

TODOs so I won't forget:

lfcnassif commented 9 months ago

Hi @costa-arc! I think I finished all logic changes, just localization to other languages is pending, I'll do later.

I would appreciate a lot if you can test my changes on your Threema file samples to test if I broke something. As I said before, this was a very complete Chat parser contribution, missing just a few minor details, thank you very much again!

lfcnassif commented 9 months ago

Just pushed a behavior change I forgot to commit, please see the updated comments in the changed code.

costa-arc commented 9 months ago

Hi @costa-arc, I finished all my changes, so I'm approving this. But I'll wait until you confirm my changes didn't messed up processing your test samples before merging. Thank you very much again!

Sure, i’ll test it today and tomorrow and post the results here.

lfcnassif commented 9 months ago

Hi @costa-arc. Since other PRs were merged, this now has a minor conflict with master. So I'm merging this right now to avoid more possible conflicts. Please let me know if something goes wrong when you have time to test my changes with your samples. Thank you very much again for this nice PR!

costa-arc commented 9 months ago

Hi @costa-arc. Since other PRs were merged, this now has a minor conflict with master. So I'm merging this right now to avoid more possible conflicts. Please let me know if something goes wrong when you have time to test my changes with your samples. Thank you very much again for this nice PR!

Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.

I did manage to test this with some of my databases and in one case the chat atachments didn't work anymore, but as soon as I have some time I will look further into this minor issue.

Anyway, it's nice to see this merged, this is a great project and i'm glad to be a small part of it.

lfcnassif commented 9 months ago

Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.

Don't worry, I understand, I have been very busy too.

in one case the chat atachments didn't work anymore

Did you see commit 7017903? Or thumbs/links to attachments stopped to work in the internal viewer too? If it is possible to share your data privately, I can try to take a look.

costa-arc commented 9 months ago

Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.

Don't worry, I understand, I have been very busy too.

in one case the chat atachments didn't work anymore

Did you see commit 7017903? Or thumbs/links to attachments stopped to work in the internal viewer too? If it is possible to share your data privately, I can try to take a look.

@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?

lfcnassif commented 9 months ago

@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?

Great! You can send me a link to a OneDrive share by our Teams work account.

lfcnassif commented 9 months ago

@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?

Just to finish this thread, @costa-arc sent me 4 Threema databases and their attachments and they were decoded properly by the final version of this PR, he thinks he had a possible build issue.