MFT Entries Carving

[wladimirleite]

IPED already includes deleted files (present in the MFT, but marked as deleted, using TSK) and has carving based on file signatures. This would be another way to recover deleted files, possibly useful in specific cases.

I started working on this idea about 3 years ago, but eventually stopped as it didn't look very promising. A few weeks ago, I started digging into it again and found a few cases in which recovered information would be useful.

The idea is to recover MFT entries (using IPED carving based on file signature) outside the MFT, when processing NTFS volumes. For recovered entries, it is possible to get:

1. File name and dates.
2. Content of very small files (< ~700 bytes), in which data is resident (i.e. stored inside the entry).
3. Non resident content.

The third case is more complex, and I still need to check if it is feasible (and useful in practice). Even if it is not possible to initially include the third feature, the first two may still be useful.

@lfcnassif, as far as I know, this kind of recovery is used by other forensic tools. Was there any attempt to implement it in the past? As I said, it may be useful in specific cases, but I still need to sort out some parts of its implementation and validate the feature in practice (using real evidences). How does that sound to you?

[lfcnassif]

Hi @tc-wleite. Although I thought about this in the past, there was no attempt to try to implement it from my side. Other tools call this metacarving (FTK) or AFAIK recover lost files (Encase). I think this is a good idea and could be very useful. If IPED default carving module could be used for it, I think it will have minimal impact in processing time. Thanks for trying to implement this!

[wladimirleite]

Thanks @lfcnassif! I will continue what I start to implement and see how it goes.

[lfcnassif]

If you need help from our side to take part of your examinations so you could dedicate time to this, please let us know.

[fmpfeifer]

I think this could be very useful. AFAIK, some commercial tools use this aproach to recover files (Runtime GatDataBack for NTFS is one example). Maybe some heuristics may be needed to find the position of the beginnign of the filesystem (Physical address of the first cluster), as those MFT entries can be from an earlier partition or from a deleted virtual machine disk file

[wladimirleite]

@lfcnassif, while carving MFT entries, I want to always discard invalid ones as there may be too many of them already. I noticed that it is already supported, through methods validateCarvedObject() and isSpecificIgnoreCorrupted() of AbstractCarver class.

A quick implementation question: I would like to use parsing code inside the validation method. Parts of that same code will be using in a parser. What is the correct way of doing that, in terms of project organization?

1. Reference iped-parsers classes from iped-carvers.
2. Place common classes in iped-utils project, and use them in iped-parsers and iped-carvers.
3. None of the above.

[lfcnassif]

I noticed that it is already supported, through methods validateCarvedObject() and isSpecificIgnoreCorrupted() of AbstractCarver class.

Yes, the isSpecificIgnoreCorrupted() could be overridden by custom implementations. A possible enhancement is to allow setting it from CarverConfig.xml, so this flag could be set based on each specific carver type.

I think the organization (1) would be better, since parsing code would fit better in parsing package. And since iped-parsers is built before iped-carvers now, I see no issue in creating this new dependency.

If parsing the new entries is a heavy operation (I guess it isn't), maybe the new items could be validated and discarded later when parsed by ParsingTask, throwing a TikaException, this would avoid duplicate parsing. But today the "discard corruped carved files" flag is global, it could be enhanced as I said above...

[wladimirleite]

Thanks @lfcnassif! Parsing is not heavy and carved entries have only 1024 bytes. I will discard them during carving and put the common classes in iped-parsers. It should be simple to reorganize it later, if we want to.

[lfcnassif]

@tc-wleite, out of curiosity, do you think your MFT entry parsing code could be enough to implement #1298 without external dependencies?

[wladimirleite]

@tc-wleite, out of curiosity, do you think your MFT entry parsing code could be enough to implement #1298 without external dependencies?

That was not my goal, as there are countless details to "fully parse the MFT". However, I can investigate it later, and hopefully implement at least a simple parser.

[lfcnassif]

Don't worry, I think #1298 is low priority.