Better event and technical metadata using IsoBuster's reporting features

bitsgalore commented 6 years ago

Starting with IsoBuster 4, reporting is possible on LOTS of things, see here:

https://www.isobuster.com/help/use_of_command_line_parameters#export-list

Also DFXML output:

https://www.isobuster.com/dfxml-example.php

(Caveat: 1 filesystem at a time, apparently).

bitsgalore commented 6 years ago

Following command line seems to work:

"C:\Program Files (x86)\Smart Projects\IsoBuster\isobuster" /d:D: /ei:test.iso /et:u /ep:oea /ep:npc /c /m /nosplash /l:test.log /tree:all:log.txt?"{'DFXML (IsoBuster 4.1 version)'}{%UTF8}{%XML}{%GMT}{%FOLDERS}{%STREAMS}<%XMLHEADER><%BR><dfxml xmlns='http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML'<%BR> xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'<%BR> xmlns:dc='http://purl.org/dc/elements/1.1/'<%BR> xmlns:hfs='http://www.forensicswiki.org/wiki/HFS' version='1.0'><%BR><%BR> <metadata><%BR> <dc:type><%DEVICETYPE></dc:type><%BR> </metadata><%BR><%BR> <creator><%BR> <program><%APP></program><%BR> <version><%VERSION></version><%BR> <execution_environment><%BR> <start_time><%SYSTIMEDATE></start_time><%BR> <os_version><%OS></os_version><%BR> <username><%USER></username><%BR> </execution_environment><%BR> </creator><%BR><%BR> <source><%BR> <device_model><%DEVICE></device_model><%BR> <image_filename><%DEVICEPATH></image_filename><%BR> <image_size><%DEVICEFILESIZE></image_size><%BR> <sectorsize><%DEVICEBLOCKSIZE></sectorsize><%BR> <devicesectors coding='base10'><%DEVICEBLOCKS></devicesectors><%BR> </source><%BR><%BR> <volume><%BR> <ftype_str><%TYPE></ftype_str><%BR> <partition_offset><%PARTITIONLBABYTESOFFSET></partition_offset>{%HEADER}{%FOLDER}<%BR> <fileobject><%BR> <filename><%RELPATH></filename><%BR> <name_type>d</name_type><%BR> <filesize><%BYTES></filesize><%BR> <alloc>1</alloc><%BR> <inode><%UID></inode><%BR> <mtime><%TIMEDATE></mtime><%BR> <byte_runs><%EXTENTLOOP> </byte_runs><%BR> </fileobject>{%FILE}<%BR> <fileobject><%BR> <filename><%RELPATH></filename><%BR> <name_type>r</name_type><%BR> <filesize><%BYTES></filesize><%BR> <alloc>1</alloc><%BR> <inode><%UID></inode><%BR> <mtime><%TIMEDATE></mtime><%BR> <hfs:HFStype_creator><%TYPE>/<%CREATOR></hfs:HFStype_creator><%BR> <byte_runs><%EXTENTLOOP> </byte_runs><%BR> </fileobject>{%STREAM}<%BR> <fileobject><%BR> <filename><%RELPATH></filename><%BR> <name_type>-</name_type><%BR> <filesize><%BYTES></filesize><%BR> <alloc>1</alloc><%BR> <inode><%UID></inode><%BR> <mtime><%TIMEDATE></mtime><%BR> <byte_runs><%EXTENTLOOP> </byte_runs><%BR> </fileobject>{%EXTENT} <byte_run img_offset='<%LBABYTEOFFSET>' len='<%BYTES>' />{%FOOTER} </volume><%BR><%BR> <runstats><%BR> <stop_time><%SYSTIMEDATE></stop_time><%BR> <clock_seconds><%SYSTIMELAPSEDSEC></clock_seconds><%BR> </runstats><%BR><%BR></dfxml><%BR>"

BUT resulting XML does not validated against dfxml schema!

bitsgalore commented 6 years ago

This POC works:

https://gist.github.com/bitsgalore/a831c6d76c097575429f3bd1aaf2a52a

Report formatting string could be defined in Iromlab's config file.

bitsgalore commented 6 years ago

DFXML reporting now works: https://github.com/KBNLresearch/iromlab/commit/5f5ce1e179b1c6524e934fbd6d9650b932fb2096

Resulting XML does not validate against DFXML XSD:

xmllint --noout isobuster-report.xml --schema ~/dfxml_schema/dfxml.xsd

Result:

Element '{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}start_time': '09/01/2018 16:12:40' is not a valid value of the atomic type 'xs:dateTime'.
Element '{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}device_model': This element is not expected. Expected is ( {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}image_filename ).
Element '{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}partition_offset': This element is not expected. Expected is one of ( {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}block_count, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}first_block, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}last_block, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}allocated_only, ##other{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}*, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}diskimageobject, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}volume, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}fileobject, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}error, ##other{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}* ).
Element '{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}runstats': This element is not expected. Expected is one of ( {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}volume, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}fileobject, {http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}rusage, ##other{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}* ).
E:/testiromlab/kb-cb8acdec-f557-11e7-b094-7446a0b42b9a/d0ae40d8-f557-11e7-b7bf-7446a0b42b9a/isobuster-report.xml fails to validate

Note on individual errors:

xs:dateTime issue

Date formatting issue. Error disappears if we change:

<start_time>10/01/2018 11:23:47</start_time>

into:

<start_time>2018-01-10T11:23:47</start_time>

See also info on dateTime format: http://validator.iatistandard.org/common_errors.php

But the IsoBuster format is defined by %SYSTIMEDATE (https://www.isobuster.com/help/use_of_command_line_parameters), not clear if there's any way to change the default?

device_model issue

Within a source element only an image_filename element is allowed. Element device_model not defined at all in DFXML. BUT this DFXML tag-library document on the BitCurator wiki does list it as part of DFXML! It is also included in this (old?) DFXML DTD by Simson Garfinkel, and in this example on Forensics Wiki.

partition_offset issue

According to schema partition_offset must be defined before ftype_str. FIXED in https://github.com/KBNLresearch/iromlab/commit/a3a7f415d4f0b0952cdb62df737693da734c9588

If we change this accordingly, xmllint produces additional errors on the mtime elements:

Element '{http://www.forensicswiki.org/wiki/Category:Digital_Forensics_XML}mtime': '27/02/2002 11:10:13' is not a valid value of the atomic type 'xs:dateTime'.

This is again because the dates are not formatted according to the dateTime format (see xs:dateTime issue above).

runstats issue

Element is not defined in DFXML; looks like this was adapted from the fiwalk output format. BUT this DFXML tag-library document on the BitCurator wiki does list it as part of DFXML!

UPDATE

The DFXML section of the report Disk Image Content Model and Metadata Analysis ACTIVITY 2: Metadata Analysis provides some more info on the confusing status of the DFXML spec. In particular:

We encountered a DTD file in our investigation (dfxml.dtd) that contains elements that do not appear in any of the schema documents (see below). The DTD can be found on Simson Garfinkel’s GitHub site and in the “Additional Tools > DFXML Scripts” tab in BitCurator. To our knowledge, the DTD is not used in the creation of DFXML files on BitCurator or via fiwalk/SleuthKit, but more investigation is necessary.

bitsgalore commented 6 years ago

Isobuster 4.2's %DFXML variable fixes the dateTime issue (and also not well-formed XML due to reserved characters, which are now replaced by entity references). Changed formatting string accordingly:

https://github.com/KBNLresearch/iromlab/commit/31f560466527677526dff9d42825361c23e8e6d8

KBNLresearch / iromlab