Support for UDF - Githubissues

bitsgalore commented 7 years ago

Reported by Eddy Colloton. We probably need rudimentary UDF support (isolyzer results are as expected for all DVDs I've come across so far, but that might be because they are using the UDF Bridge format (hybrid ISO9660/UDF FS).

Original issue turned out to be unrelated to the UDF fs, BUT UDF support might still be useful.

UDF spec (2005 revision):

http://www.osta.org/specs/pdf/udf260.pdf

Page 24, section 2.2.4 Logical Volume Descriptor, field LogicalBlockSize.
Then see page 26, section 2.2.6 Logical Volume Integrity Descriptor:

Through the contents of the Logical Volume Integrity Descriptor an implementation can easily answer the following useful questions: (...) 4) What is the total size of the Logical Volume in logical blocks?

Which looks pretty much like what is needed to do the verification for UDF.

bitsgalore commented 7 years ago

First question: do we really need UDF support if most/all DVDs have UDF Bridge layout? Related standards docs (all paywalled):

https://ansidotorg.blogspot.nl/2016/04/iso-image-and-udf-disk-image.html

UDF bridge spec:

https://www.ecma-international.org/publications/files/ECMA-TR/ECMA%20TR-071.PDF

Almost all DVD-Video discs use the UDF bridge format

https://en.wikipedia.org/wiki/DVD-Video

(But: most recent source cited dates back to 2011!).

DVD-Video and DVD-Audio use UDF 2.0 exclusively, with no redundant ISO 9660 file system.

http://www.afterdawn.com/glossary/term.cfm/udf_bridge

DVD-ROM discs use the UDF Bridge format.

http://www.blazevideo.com/blog/what-is-udf-and-udfiso%EF%BC%9F.html

BUT below link suggests UDF-only DVDs do exist in the wild:

https://askubuntu.com/questions/359264/how-to-open-udf-volume

Wenguang's Introduction to Universal Disk Format (UDF):

https://sites.google.com/site/udfintro/

Steps (I think?):

Locate Logical Volume Descriptor (LVD)
From LVD, read logical block size
From LVD, read Integrity Sequence Extent --> contains sector address of Logical Volume Integrity Descriptor (LVID), analogous to ExtentLength in existing Py code.
From LVID, read NumberOfPartitions
From LVID, read SizeTable, which is described as follows in ECMA-167:

This field shall contain N_P (= Number of Partitions) values, each recorded as a Uint32, recorded contiguously starting at the first byte of the field. The ith value specifies the size, in logical blocks, of the partition specified by the ith entry in the Partition Maps field in the associated Logical volume Descriptor. A value of #FFFFFFFF shall mean that the size is not specified.

See also ECMA-167, which is more informative than UDF spec here.

bitsgalore commented 7 years ago

Creating a UDF file on Linux:

https://unix.stackexchange.com/a/17613/187090

Create empty image:

truncate -s 2M test.udf

Add file system:

mkudffs --media-type=dvd test.udf

Output:

start=0, blocks=16, type=RESERVED 
start=16, blocks=3, type=VRS 
start=19, blocks=237, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=493, type=PSPACE 
start=767, blocks=1, type=ANCHOR 
start=768, blocks=239, type=USPACE 
start=1007, blocks=16, type=RVDS 
start=1023, blocks=1, type=ANCHOR

From Isolyzer output for this image:

<sizeExpected>1009664</sizeExpected>
<sizeActual>2097152</sizeActual>
<sizeDifference>1087488</sizeDifference>
<sizeAsExpected>False</sizeAsExpected>
<smallerThanExpected>False</smallerThanExpected>

So image 1087488/2048 = 531 sectors smaller than expected!

Reported properties:

<properties>
    <udf>
        <logicalVolumeDescriptor>
            <tagIdentifier>6</tagIdentifier>
            <descriptorVersion>3</descriptorVersion>
            <tagSerialNumber>1</tagSerialNumber>
            <volumeSequenceNumber>2</volumeSequenceNumber>
            <logicalVolumeIdentifier>LinuxUDF</logicalVolumeIdentifier>
            <logicalBlockSize>2048</logicalBlockSize>
            <domainIdentifier>*OSTA UDF Compliant</domainIdentifier>
            <mapTableLength>6</mapTableLength>
            <numberOfPartitionMaps>1</numberOfPartitionMaps>
            <implementationIdentifier>*Linux UDFFS</implementationIdentifier>
            <integritySequenceExtentLength>2048</integritySequenceExtentLength>
            <integritySequenceExtentLocation>273</integritySequenceExtentLocation>
        </logicalVolumeDescriptor>
        <logicalVolumeIntegrityDescriptor>
            <tagIdentifier>9</tagIdentifier>
            <descriptorVersion>3</descriptorVersion>
            <tagSerialNumber>1</tagSerialNumber>
            <timeStamp>2017/06/29, 13:22:39</timeStamp>
            <integrityType>1</integrityType>
            <numberOfPartitions>1</numberOfPartitions>
            <lengthOfImplementationUse>46</lengthOfImplementationUse>
            <freeSpaceTable>488</freeSpaceTable>
            <sizeTable>493</sizeTable>
        </logicalVolumeIntegrityDescriptor>
    </udf>

Note freeSpaceTable value of 488 sectors (expected 0). Also note that the sizeTable value corresponds to PSPACE in mkudffs output above.

bitsgalore commented 7 years ago

UDF file size check now implemented https://github.com/KBNLresearch/isolyzer/commit/e8982006d4282d122453332cbd836dd676408842

BUT the file size estimate from the UDF headers is always 263-265 sectors short on the real file size (even in cases when ISO PVD fields give exact fit on actual file size). So perhaps the UDF-based estimate doesn't cover the whole file but starts some sectors into the image?

KBNLresearch / isolyzer

Support for UDF #5