search

libyal / libewf

Libewf is a library to access the Expert Witness Compression Format (EWF)
GNU Lesser General Public License v3.0
263 stars 76 forks source link

reading +4G size ltree hits allowed memory limit #163

Closed mikebbt closed 1 year ago

mikebbt commented 2 years ago

I'm attempting to read a very large L01 file (490+GB), the ltree is approximately 9.7+GB of that total. I'm encountering an error in libcfile_file_read_buffer_with_error_code(), "invalid size value exceeds maximum" when reading the ltree section.

This only occurs on Windows (same L01 is read by libewf on MacOS w/o issue), I see the defines to adjust the check for size (snippet below) but not sure what the intent was, I assume to put a ceiling on the max size the buffer ever could be? Unless I've missed something we will always test for (size > (size_t)UINT32_MAX) and in my case it will always fail for any ltree larger than 4294967295.

if ( UINT32_MAX < SSIZE_MAX )

if( size > (size_t) UINT32_MAX )

else

if( size > (size_t) SSIZE_MAX )

endif

Assuming the L01 doesn't have some other issue, was libewf ever intended to read something this big?

Thanks mikebbt

joachimmetz commented 2 years ago

Assuming the L01 doesn't have some other issue, was libewf ever intended to read something this big?

Not for L01s. Since it is a propriety format that is not documented. So you're hitting a format sanity check. Also a data format that requires having to read +4G buffer in memory for a file hierarchy is not very efficient.

mikebbt commented 2 years ago

Just curious why Mac doesn't have a similar sanity check?

joachimmetz commented 2 years ago

Just curious why Mac doesn't have a similar sanity check?

Has probably nothing to do with Mac, I assume SSIZE_MAX is larger

joachimmetz commented 1 year ago

Try a recent version I switched to a line reader approach, such large logical file trees might still consume a lot of memory.