search

libyal / libewf-legacy

Legacy version of libewf
GNU Lesser General Public License v3.0
10 stars 5 forks source link

ewfaquire libcdata_internal_array_resize: invalid entries size value exceeds maximum #9

Closed rwiesbach closed 3 years ago

rwiesbach commented 3 years ago

Version 20140808 installed via brew

command ewfaquire -t %destpath% -c fast /dev/rdisk5 Error message: Unable to aquire input. libcdata_internal_array_resize: invalid entries size value exceeds maximum. ... Unable to close output file(s). libcdata_internal_array_resize: invalid entries size value exceeds maximum. ...

(shortened as I need to type it from screen of a airgapped system, let me know if you need more lines.)

functional dd version: dd if=/dev/rdisk5 bs=1m | ewfaquirestream -t %destpath% -c fast

rdisk6, which is synthesized from disk 6, works. Maybe the "synthesization" is cause of the issue?

joachimmetz commented 3 years ago

Moving to https://github.com/libyal/libewf-legacy

joachimmetz commented 3 years ago

rdisk6, which is synthesized from disk 6, works. Maybe the "synthesization" is cause of the issue?

"libcdata_internal_array_resize: invalid entries size value exceeds maximum." indicates a limit is being reached.

rwiesbach commented 3 years ago

Sorry for issue in the wrong repo.

Mac OS 10.14.5 Mojave about 1TB

full trace, as written before, will take a while to transfer from the airgapped system, hoped that would not be necessary.

joachimmetz commented 3 years ago

ack, I think you might be hitting a hard coded limit with 1 TB / 1 TiB. Does the following trace back look very similar (in particular the libewf functions)?

ewfverify 20140808

Unable to open EWF image file(s).
libcdata_internal_array_resize: invalid entries size value exceeds maximum.
libcdata_array_resize: unable to resize array.
libmfdata_list_resize: unable to resize elements array.
libewf_segment_file_read_volume_section: unable to resize chunk table list.
libewf_handle_open_read_segment_files: unable to read section: volume.
libewf_handle_open_file_io_pool: unable to read segment files.
libewf_handle_open: unable to open handle using a file IO pool.
verification_handle_open_input: unable to open files.
rwiesbach commented 3 years ago

hardcoded limit? I definitely used ewfacquire for > 1TB before (not on MacOS, though)

I had to find out that the acquirestream workaround fails at 511GB (input):

ewfacquirestream after 511GB: Status: acquired 511 GiB (549718261760 bytes)

    in 2 hour(s), 3 minute(s) and 37 second(s) with 70 MiB/s (74115985 bytes/second).

Acquiry failed at: Oct 21, 2020 19:51:16 Unable to read input. libcdata_internal_array_resize: invalid entries size value exceeds maximum. libcdata_array_resize: unable to resize array. libmfdata_list_resize: unable to resize elements array. libewf_segment_file_write_chunks_section_start: unable to resize chunk table list. libewf_handle_write_buffer: unable to write new chunk. imaging_handle_write_buffer: unable to write storage media buffer. ewfacquirestream_read_input: unable to write data to file.

ewfacquire Acquiry failed at: Oct 21, 2020 19:57:14 Unable to acquire input. libcdata_internal_array_resize: invalid entries size value exceeds maximum. libcdata_array_resize: unable to resize array. libmfdata_list_resize: unable to resize elements array. libewf_write_io_handle_write_new_chunk: unable to resize chunk table. libewf_handle_write_buffer: unable to write new chunk. imaging_handle_write_buffer: unable to write storage media buffer. ewfacquire_read_input: unable to write data to file. Unable to close output file(s). libcdata_internal_array_resize: invalid entries size value exceeds maximum. libcdata_array_resize: unable to resize array. libmfdata_list_resize: unable to resize elements array. libewf_write_io_handle_write_new_chunk: unable to resize chunk table. libewf_handle_write_finalize: unable to write new chunk. libewf_handle_close: unable to finalize write. imaging_handle_close: unable to close output handle.

joachimmetz commented 3 years ago

hardcoded limit?

based on the trace back I would say https://github.com/libyal/libcdata/blob/583e22d7962ec7c09724a9c8b66aecf44aa6e59c/libcdata/libcdata_array.c#L782

but need to take a closer look at the libcdata version provided in the 20140808 package and how brew is building libewf.

rwiesbach commented 3 years ago

Thank you. Is that a limitation of libc, or a limitation of my workstations memory? Is there a quick fix?

joachimmetz commented 3 years ago

Is that a limitation of libc, or a limitation of my workstations memory? Is there a quick fix?

a limit of libcdata (which is not libc), the quick fix is increasing the limit and recompiling. I'll see if I can reproduce this and maybe make some changes to increase the limit (with still having a safe guard against sporadic memory increase).

rwiesbach commented 3 years ago

Did you find your time for testing? Do you have an idea of an appropriate larger value for LIBCDATA_ARRAY_ENTRIES_MEMORY_LIMIT that I can try?

joachimmetz commented 3 years ago

Did you find your time for testing?

Not yet, have a couple of other priorities on my plate at the moment.

Do you have an idea of an appropriate larger value for LIBCDATA_ARRAY_ENTRIES_MEMORY_LIMIT that I can try?

It's 128 MiB at the moment if you can afford it, you can try to up the limit to 1 GiB or more

rwiesbach commented 3 years ago

Thank you. Do I assume correctly that the value is proportional and if a config of 128MiB fails at 511GB, then 256 MB should be enough for a image size of 1TB? How does memory usage scale, is this just a unique data structure or a per-thread datastructure or similar?

joachimmetz commented 3 years ago

depends on the chunk size, approx 256 MiB for a 32 KiB chunk, I've made some changes in https://github.com/libyal/libewf-legacy/commit/bd01ca2e2084ea1b3ff02ef4b8bda6dddfa49935 that should allow for < 4 TiB with a 32 KiB chunk size

joachimmetz commented 3 years ago

Closing issue, reopen if still an issue.

cnc24 commented 2 years ago

Where do I change the LIBCDATA_ARRAY_ENTRIES_MEMORY_LIMIT Value, because the problem still exists in version 20140807. I want to aqcuire some NVME devices and i stuck on this error.

joachimmetz commented 2 years ago

@cnc24 20140807 is very old (Mar 17, 2019) please use a more recent version

cnc24 commented 2 years ago

https://packages.debian.org/sid/admin/ewf-tools

It’s the recent version at the Debian repository, Do you have an more recent repo for Ubuntu or Debian?

joachimmetz commented 2 years ago

@cnc24 the version in the Debian version/repository (and Ubuntu version for that matter) is maintained by the Debian maintainers, not this project. You'll need to reach out to them.

JustMyGithub commented 1 year ago

Seems like I have the same error with a current build on Cygwin at ~ 4TB. It happened with ewfacquirestream with -c best as well as with -c fast

Status: acquired 3.9 TiB (4397509784736 bytes)
        in 17 hour(s), 1 minute(s) and 52 second(s) with 68 MiB/s (71723476 bytes/second).

Acquiry failed at: Jan 15, 2023 XX:XX:XX
Unable to read input.
libmfdata_array_resize: invalid entries size value exceeds maximum.
libmfdata_list_resize: unable to resize elements array.
libewf_segment_file_write_chunks_section_start: unable to resize chunk table list.
libewf_handle_write_buffer: unable to write new chunk.
imaging_handle_write_buffer: unable to write storage media buffer.
ewfacquirestream_read_input: unable to write data to file.
Unable to close output.
libewf_section_start_read: checksum does not match (stored: 0x00000006 calculated: 0x5ac1085a).
libewf_segment_file_read: unable to read section start.
libmfdata_file_list_get_file_value: unable to read file data from pool entry: 72.
libmfdata_file_list_get_file_value_by_index: unable to retrieve file value.
libewf_write_io_handle_finalize_write_sections_corrections: unable to retrieve segment file: 73 from list.
libewf_handle_write_finalize: unable to write sections corrections to segment files.
libewf_handle_close: unable to finalize write.
imaging_handle_close: unable to close output handle.