Changes for reading LEF files

[jtsylve]

This rather large PR adds improved support for handling the reading of LEF (L01) files

[jtsylve]

We're looking into the test_write_functions CI failure

[joachimmetz]

@jtsylve can you be more specific what theses changes fix? Also what is the relationship between this PR and the 2 PRs closes without a comment?

[codecov[bot]]

Codecov Report

Merging #132 into master will decrease coverage by 5.28%. The diff coverage is 2.31%.

@@            Coverage Diff             @@
##           master     #132      +/-   ##
==========================================
- Coverage   33.99%   28.71%   -5.29%     
==========================================
  Files          59       70      +11     
  Lines       26364    32448    +6084     
==========================================
+ Hits         8962     9316     +354     
- Misses      17402    23132    +5730

Impacted Files	Coverage Δ
libewf/libewf_single_files.c	2.26% <ø> (-2.61%)	:arrow_down:
libewf/libewf_file_source.c	0% <ø> (ø)
libewf/libewf_single_file_source.c	27.27% <ø> (ø)
libewf/libewf_file_extended_attribute.c	0% <0%> (ø)
libewf/libewf_file_entry.c	0.49% <0%> (-0.17%)	:arrow_down:
libewf/libewf_file_subject.c	0% <0%> (ø)
libewf/libewf_single_file_permission_tree.c	0% <0%> (ø)
libewf/libewf_single_file_subject.c	0% <0%> (ø)
libewf/libewf_single_file_extended_attribute.c	0% <0%> (ø)
libewf/libewf_file_permission.c	0% <0%> (ø)
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d443f5a...bacf57d. Read the comment docs.

[jtsylve]

If you need more specifics, I'll have to ask the person who worked on this to comment, but basically this PR enhances libewf's support for LEF files in a number of ways.

1) It exposes new functionality to the API for querying a number of new metadata properties of stored logical files

2) It fixes issues with reading certain standards compliant LEF files that read fine in Encase and FTK, but seem to break tools that rely on libewf.

The two PRs that I closed were part of my failed attempt at breaking this larger PR into several smaller PRs, that can be ignored.

[kastonework]

Detail on the changes: The majority of the changes are additions to libewf to support reading of additional single file entry data from LEFs (guid, fileID, deletedDate, record type, sources tree, subjects tree, permissions tree, and extended attributes tree).

The other change was a fix for a bug identified in libewf:

• Fix issue where streamed data was truncated if it was not a multiple of the ev file sector size (could be reproduced via ewfacquirestream).

There had been another bug fix in there, but that was causing the CI test failure so it was reverted. That issue (missing number_of_chunks and number_of_sectors in data section of ev files with >64 segments) will likely get reported as a bug with more detail.

[joachimmetz]

@kastonework @jtsylve seeing this PR changes a lot of code and differs from the code style it will take significant time to get the changes merged. And my time is very limited at the moment.

It fixes issues with reading certain standards compliant LEF files that read fine in Encase and FTK, but seem to break tools that rely on libewf.

Can you provide more detail or test files. It is important that differences in the data format are also documented. Since the "standard" is proprietary.

[kastonework]

... and differs from the code style it will take significant time to get the changes merged.

The intention was to match the existing code style; if you can point out an example of differences then I will update the new code to match.

[jtsylve]

@joachimmetz Is there a clang-format (or similar tool) style that can be applied to automatically format the code for consistency?

[joachimmetz]

Is there a clang-format (or similar tool) style that can be applied to automatically format the code for consistency?

unfortunately not, but no worries, I mainly mentioned this to indicate that merging will take some time.

What I do would like to know is:

It fixes issues with reading certain standards compliant LEF files that read fine in Encase and FTK, but seem to break tools that rely on libewf.

Can you provide more detail or test files. It is important that differences in the data format are also documented. Since the "standard" is proprietary.

[joachimmetz]

Leaving comments FYI and notes to self while preparing the code for merge.

[joachimmetz]

@jtsylve to better understand this large PR and to ensure there is test coverage and not to merge unused code I'll make similar changes based on PR but not necessarily the same.

[joachimmetz]

@jtsylve @kastonework @mikebbt if one of you has sample files with multiple LEF sources and/or actual subjects (not the defaults ones) that would be helpful. I do not necessarily need the whole file, just the ltree (single files) serialized object data (string).

[jtsylve]

I didn't actually do the work in these patches. I'll ask @kastonework to respond.

[joachimmetz]

Ack, then I should credit her instead.

[kastonework]

@joachimmetz @jtsylve I’ll review and respond to all of the questions in the next day or so.

[joachimmetz]

@kastonework no hurries, I'll keep working on adding equivalent functionality. I'll ask you to double check if everything is there when I'm finished. I'm deviating from your original approach in some places.

What could be useful in the intermediate time is having specific test data:

• single files with multiple sources
• single files one or more subjects

[jtsylve]

Ack, then I should credit her instead.

Yes, Please credit Kim and BlackBag for the work. I just put in the PR.

[kastonework]

@jtsylve @kastonework @mikebbt if one of you has sample files with multiple LEF sources and/or actual subjects (not the defaults ones) that would be helpful. I do not necessarily need the whole file, just the ltree (single files) serialized object data (string).

I can provide this; should I email it to you?

[joachimmetz]

I can provide this; should I email it to you?

@kastonework yes please

[joachimmetz]

was auto closed due to migration to main branch, will look into merging parts of the PR that will be beneficial