search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
949 stars 219 forks source link

Upgrade Apache Commons Compress #1961

Open wladimirleite opened 11 months ago

wladimirleite commented 11 months ago

Processing a 7z of ~7 GB, another user here got several errors. Testing the archive (with the 7-zip program), it seems fine. After investigating other possibilities, I tried to use the latest version of Apache Commons Compress (1.24.0), and the processing finished successfully (currently, version 1.21 is used).

@lfcnassif, I know that upgrading this kind of dependency is critical, so I will leave this issue as a suggestion to be evaluated.

lfcnassif commented 11 months ago

Hi @tc-wleite! I think we can upgrade this and other dependencies on master, targeting 4.2.0. I already plan to upgrade Tika for 4.2.0 and I should run a large regression test before releasing it.

wladimirleite commented 7 months ago

I noticed that #2091 pointed out a possible vulnerability in 1.21, recommending the upgrade to 1.26.

lfcnassif commented 7 months ago

There are 2 fixed CVEs in 1.26, CVE-2024-26308 (OOME) is worse to me, let's upgrade. Sorry @wladimirleite, I missed this in the queue.

lfcnassif commented 7 months ago

Reopening, while testing #2093, I detected in a small test a lot of .ar (application/x-archive) files stopped to be expanded. I'll test Compress 1.24 and 1.25 to see if that .ar regression disappears, but I think we should do a large test before this upgrade.

lfcnassif commented 7 months ago

CommonsCompress-1.25 doesn't have the *.ar regression I detected. I'll downgrade to it on master, but I'll leave this open until we run a larger test to see if there are other regressions.

wladimirleite commented 7 months ago

Reopening, while testing https://github.com/sepinf-inc/IPED/pull/2093, I detected in a small test a lot of *.ar (application/x-archive) files stopped to be expanded.

That is bad news.

CommonsCompress-1.25 doesn't have the *.ar regression I detected. I'll downgrade to it on master, but I'll leave this open until we run a larger test to see if there are other regressions.

Are the vulnerabilities solved in 1.25?

lfcnassif commented 7 months ago

Are the vulnerabilities solved in 1.25?

I think no. Although non ideal, we have timeouts to protect against infinite loops (DoS) and external parsing to isolate OOMEs. Fortunately there is no (known) Remote Code Execution vulnerability.

lfcnassif commented 7 months ago

Hopefully

I meant Fortunately.

wladimirleite commented 7 months ago

If the only problem with 1.26 is with AR, maybe one option would be using SevenZipParser instead of PackageParser.

lfcnassif commented 7 months ago

If the only problem with 1.26 is with AR, maybe one option would be using SevenZipParser instead of PackageParser.

That worked!

wladimirleite commented 2 weeks ago

@lfcnassif, working on a real case, very important valid ZIP files were not expanded. Fortunately, I noticed the parser exceptions and tried to figure out what was going on. Using the latest version of common-compress (1.27.1) solved the issue. I know that there are a lot of things to deal with before releasing 4.2, but upgrading this library seems very important.

lfcnassif commented 2 weeks ago

Hi @wladimirleite, thank you for reporting that. ZIP is much more important to handle correctly than AR. I'll upgrade when I have some time. But could you test the original 7zip sample referenced by the first post here with commons compress 1.27.1?

wladimirleite commented 2 weeks ago

But could you test the original 7zip sample referenced by the first post here with commons compress 1.27.1?

Sure! I will try with that 7z.

lfcnassif commented 2 weeks ago

One question: does 4.1.6 version handle those valid zip files correctly?

wladimirleite commented 2 weeks ago

One question: does 4.1.6 version handle those valid zip files correctly?

No, it doesn't. I tested the 7z I mentioned when I opened this issue using common compress 1.27.1, and it worked fine. However, I found other ZIPs (from the some source I mentioned yesterday), which still fail using 1.27.1 (they also fail using previous versions). Not sure what is causing this, as they are opened with other tools. I am investing it a bit further, and will update here.