Closed costa-arc closed 9 months ago
Thank you @costa-arc for this Threema parser contribution!
Hi @patrickdalla, I'm changing the reviewer of this to me and assigning #1956 to you when you finish the other ones, thanks!
@costa-arc, sorry for our delay here. I'll start a crawling for Threema database samples in our 500 cases database so we can test this properly.
The crawling finished and after scanning ~800 cases, I just found Threema databases in 2 cases... @tc-wleite, when you have some time, could you search for Threema databases in São Paulo cases database? I used the search query below:
name:(threema ThreemaData) && type:(plist sqlite)
@tc-wleite, when you have some time, could you search for Threema databases in São Paulo cases database?
Sure! Just start a search here.
I just reverted some unrelated changes and pushed.
Sure! Just start a search here.
Thanks @tc-wleite! When it finishes, you can send the samples to me by Teams, thanks!
Thanks @tc-wleite! When it finishes, you can send the samples to me by Teams, thanks!
Unfortunately, I found Threema files in only 2 cases (out of 2,000). And by the length of the files, they don't seem to contain much information (if any). Anyway, I just sent them to you by Teams.
Unfortunately, I found Threema files in only 2 cases (out of 2,000). And by the length of the files, they don't seem to contain much information (if any). Anyway, I just sent them to you by Teams.
Thank you! Those I got are bigger, one has 1.5MB and other 68MB.
Hi @costa-arc, I'm reviewing this PR, it is well written and has most features of other chat parsers, thank you very much! I'm just taking care of a few details.
One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.
One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.
I just saw it is possible to be outside the database if data field is 36 bytes size, which means it's the external file name. Let me know if I misunderstood.
One question: media files attached to chats are always into the chat database or is it possible to find them outside the DB? I saw you extract media files from the DB and in a second pass search for them into the case to link to chats. But if they are always stored into the DB, I think that could be done in a single pass, without the need to search for them into the case later.
I just saw it is possible to be outside the database if data field is 36 bytes size, which means it's the external file name. Let me know if I misunderstood.
Exactly @lfcnassif, larger files are stored outside the database and checking data field is the best way to find out.
TODOs so I won't forget:
Hi @costa-arc! I think I finished all logic changes, just localization to other languages is pending, I'll do later.
I would appreciate a lot if you can test my changes on your Threema file samples to test if I broke something. As I said before, this was a very complete Chat parser contribution, missing just a few minor details, thank you very much again!
Just pushed a behavior change I forgot to commit, please see the updated comments in the changed code.
Hi @costa-arc, I finished all my changes, so I'm approving this. But I'll wait until you confirm my changes didn't messed up processing your test samples before merging. Thank you very much again!
Sure, i’ll test it today and tomorrow and post the results here.
Hi @costa-arc. Since other PRs were merged, this now has a minor conflict with master. So I'm merging this right now to avoid more possible conflicts. Please let me know if something goes wrong when you have time to test my changes with your samples. Thank you very much again for this nice PR!
Hi @costa-arc. Since other PRs were merged, this now has a minor conflict with master. So I'm merging this right now to avoid more possible conflicts. Please let me know if something goes wrong when you have time to test my changes with your samples. Thank you very much again for this nice PR!
Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.
I did manage to test this with some of my databases and in one case the chat atachments didn't work anymore, but as soon as I have some time I will look further into this minor issue.
Anyway, it's nice to see this merged, this is a great project and i'm glad to be a small part of it.
Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.
Don't worry, I understand, I have been very busy too.
in one case the chat atachments didn't work anymore
Did you see commit 7017903? Or thumbs/links to attachments stopped to work in the internal viewer too? If it is possible to share your data privately, I can try to take a look.
Hi @lfcnassif, sry I didn't reply earlier, I'm having some busy weeks lately.
Don't worry, I understand, I have been very busy too.
in one case the chat atachments didn't work anymore
Did you see commit 7017903? Or thumbs/links to attachments stopped to work in the internal viewer too? If it is possible to share your data privately, I can try to take a look.
@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?
@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?
Great! You can send me a link to a OneDrive share by our Teams work account.
@lfcnassif Sure, I can share with you (privately) the data I used for testing! How can I do it?
Just to finish this thread, @costa-arc sent me 4 Threema databases and their attachments and they were decoded properly by the final version of this PR, he thinks he had a possible build issue.
Currently working for IOS filesystem (main database: ThreemaData.sqlite) Recovering deleted data from sqlite and tests for this parser not implemented yet