storaged-project / udisks

The UDisks project provides a daemon, tools and libraries to access and manipulate disks, storage devices and technologies.
https://storaged.org/doc/udisks2-api/latest/
Other
346 stars 142 forks source link

recognize multisession UDF optical discs (CD / DVD) #836

Open md-work opened 3 years ago

md-work commented 3 years ago

Please add support to recognize UDF multisession optical discs.  

udisks currently doesn't seem to recognize UDF multisession optical discs (CD / DVD). Tested with openSUSE-15.2 (udisks2-2.8.1) and Debian-11-Testing (Bullseye 2021-01-14 - udisks2-2.9.1)

Instead udisks either recognizes the disc as ISO-9660 (if an ISO-9660 filesystem exists) or simply detects no filesystem at all.    

Details

To be precise: It must be detected what filesystem the LAST session of a disc has. (for example blkid from util-linux instead somehow always detects the filesystem of the first session)

https://github.com/pali/udftools seems to have a working implementation to detect if the last session has an UDF filesystem. Just run udfinfo or udflabel on a disc.    

Reproduction

I made some CD images to reproduce the problem. udisks_udf_multisession_support.zip

Sadly multisession CDs can't simply be stored in an dd/ISO image. Instead I've created some images which can be used via CDemu (Wikipedia). Each image consists of a .bin and a .toc file for each session. Loading a full CD with all sessions works like this:

cdemu  load  0  image_session-1.toc  image_session-2.toc
md-work commented 3 years ago

If I mount the device another bug is triggered on the dbus interface. mount -o ro /dev/sr0 /mnt Usually there would be a PropertiesChanged signal on the dbus interface. But instead there's an InterfacesAdded signal.

And on unmount this bug continues. umount /mnt Usually this would trigger another PropertiesChanged signal in the dbus interface. But instead now there's an InterfacesRemoved.

tbzatek commented 3 years ago

You can assume that both the InterfacesAdded and PropertiesChanged signals are valid and can be received in random order, as a result of asynchronous nature of the messaging chain. What's your use case here?

As for the multisession, udisks relies on information from udev which in turn often relies on information from blkid. In case blkid is wrong, please open a bugreport on https://github.com/karelzak/util-linux/ and udisks should pick the information up automatically.

tbzatek commented 3 years ago

Also, could you post udevadm info /dev/....., blkid and lsblk dumps here related to the block device carrying the UDF filesystem please?

md-work commented 3 years ago

@tbzatek

You can assume that both the InterfacesAdded and PropertiesChanged signals are valid and can be received in random order, as a result of asynchronous nature of the messaging chain. What's your use case here?

I'm listening to these signals in a software which needs to know the currently connected storage devices. Don't get me wrong. In general this bug affects probably every software using udisks. So for example KDE/Plasma isn't able to mount UDF multisession discs.

I can workaround this behavior in my software. But I'm mentioning this, because the behavior differs between UDF-multisession discs and non UDF/multisession discs (everything is fine for these). So I'm considering this to be a bug in udisks and I guess it should better be fixed in udisks (especially the wrongly InterfacesRemoved signal on umount).

As for the multisession, udisks relies on information from udev which in turn often relies on information from blkid.

As far as I know blkid doesn't provide information about mounts. And mounts clearly play a role here. But I'll talk about blkid later on.

 

Though I guess this might be subsequent faults of the initially described problem. So let's not focus on this signaling problem for the moment.

 

 


 

 

Also, could you post udevadm info /dev/....., blkid and lsblk dumps here related to the block device carrying the UDF filesystem please?

Sure! (see below)

Nevertheless, please feel free to run your own tests with the provided disc images and CDemu. It should be at least reproducible with openSUSE-15.2 and Debian-11-Testing (Bullseye 2021-01-14).

I've reproduced the same results with a real DVD instead of a virtual cdemu disc. Sadly the only way to create pure UDF multisession discs I know is CDBurnerXP for Windows. So these cdemu compatible images should be fine for debugging.

Everything done as root for simplicity of testing. Prerequisite: Load & start the cdemu kernel moduel and daemon modprobe vhba; cdemu-daemon

 

used programs

All by openSUSE-15.2, except udfinfo which was compiled from version 2.3 source.

used program details ``` $ cat /etc/os-release NAME="openSUSE Leap" VERSION="15.2" ID="opensuse-leap" ID_LIKE="suse opensuse" VERSION_ID="15.2" PRETTY_NAME="openSUSE Leap 15.2" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:leap:15.2" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org/" $ cdemu version Library version: 3.2.1 Daemon version: 3.1.0 $ rpm -qf /usr/bin/udisksctl udisks2-2.8.1-lp152.1.4.x86_64 $ rpm -qf /usr/bin/udevadm udev-234-lp152.31.19.1.x86_64 $ lsblk --version lsblk from util-linux 2.33.1 $ udfinfo --help udfinfo from udftools 2.3 Usage: udfinfo [--locale|--u8|--u16|--utf8] [-b|--blocksize=block-size] [--startblock=block] [--lastblock=block] [--vatblock=block] device ```

 

 

UDF non-multisession example (everything's fine)

Loading session 1 only, so this isn't multisession.

non-multisession dump (working fine) ``` $ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc $ udisksctl info -b /dev/sr1 /org/freedesktop/UDisks2/block_devices/sr1: org.freedesktop.UDisks2.Block: Configuration: [] CryptoBackingDevice: '/' Device: /dev/sr1 DeviceNumber: 2817 Drive: '/org/freedesktop/UDisks2/drives/CDEmu_Virt_2e_CD_2fDVD_ROM' HintAuto: true HintIconName: HintIgnore: false HintName: HintPartitionable: false HintSymbolicIconName: HintSystem: false Id: by-uuid-2010112200000000 IdLabel: UDF-MulSess_1 IdType: udf IdUUID: 2010112200000000 IdUsage: filesystem IdVersion: 1.02 MDRaid: '/' MDRaidMember: '/' PreferredDevice: /dev/sr1 ReadOnly: true Size: 1249280 Symlinks: /dev/disk/by-label/UDF-MulSess_1 /dev/disk/by-path/platform-vhba-scsi-0:0:1:0 /dev/disk/by-uuid/2010112200000000 UserspaceMountOptions: org.freedesktop.UDisks2.Filesystem: MountPoints: Size: 0 $ udevadm info /dev/sr1 P: /devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1 N: sr1 S: disk/by-label/UDF-MulSess_1 S: disk/by-path/platform-vhba-scsi-0:0:1:0 S: disk/by-uuid/2010112200000000 E: DEVLINKS=/dev/disk/by-label/UDF-MulSess_1 /dev/disk/by-path/platform-vhba-scsi-0:0:1:0 /dev/disk/by-uuid/2010112200000000 E: DEVNAME=/dev/sr1 E: DEVPATH=/devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1 E: DEVTYPE=disk E: ID_BUS=scsi E: ID_CDROM=1 E: ID_CDROM_CD=1 E: ID_CDROM_CD_R=1 E: ID_CDROM_CD_RW=1 E: ID_CDROM_DVD=1 E: ID_CDROM_DVD_PLUS_R=1 E: ID_CDROM_DVD_R=1 E: ID_CDROM_DVD_RAM=1 E: ID_CDROM_MEDIA=1 E: ID_CDROM_MEDIA_CD=1 E: ID_CDROM_MEDIA_SESSION_COUNT=1 E: ID_CDROM_MEDIA_TRACK_COUNT=1 E: ID_CDROM_MEDIA_TRACK_COUNT_DATA=1 E: ID_CDROM_MRW=1 E: ID_CDROM_MRW_W=1 E: ID_FOR_SEAT=block-platform-vhba-scsi-0_0_1_0 E: ID_FS_LABEL=UDF-MulSess_1 E: ID_FS_LABEL_ENC=UDF-MulSess_1 E: ID_FS_TYPE=udf E: ID_FS_USAGE=filesystem E: ID_FS_UUID=2010112200000000 E: ID_FS_UUID_ENC=2010112200000000 E: ID_FS_VERSION=1.02 E: ID_MODEL=Virt._CD_DVD-ROM E: ID_MODEL_ENC=Virt.\x20CD\x2fDVD-ROM E: ID_PATH=platform-vhba-scsi-0:0:1:0 E: ID_PATH_TAG=platform-vhba-scsi-0_0_1_0 E: ID_REVISION=1.10 E: ID_SCSI=1 E: ID_SCSI_INQUIRY=1 E: ID_TYPE=cd E: ID_VENDOR=CDEmu E: ID_VENDOR_ENC=CDEmu\x20\x20\x20 E: MAJOR=11 E: MINOR=1 E: SCSI_MODEL=Virt._CD/DVD-ROM E: SCSI_MODEL_ENC=Virt.\x20CD/DVD-ROM E: SCSI_REVISION=1.10 E: SCSI_TPGS=0 E: SCSI_TYPE=cd/dvd E: SCSI_VENDOR=CDEmu E: SCSI_VENDOR_ENC=CDEmu\x20\x20\x20 E: SUBSYSTEM=block E: SYSTEMD_MOUNT_DEVICE_BOUND=1 E: TAGS=:systemd:uaccess:seat: E: USEC_INITIALIZED=76936869 $ blkid /dev/sr1 /dev/sr1: UUID="2010112200000000" LABEL="UDF-MulSess_1" TYPE="udf" $ lsblk /dev/sr1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr1 11:1 1 1,2M 0 rom $ ~/opt/udftools/bin/udfinfo /dev/sr1 filename=/dev/sr1 label=UDF-MulSess_1 uuid=2010112200000000 lvid=UDF-MulSess_1 vid=UDF-MulSess_1 vsid= fsid=UDF-MulSess_1 fullvsid=20101122 owner= organization= contact= appid= impid=*UDF Image Creator winserialnum=0xbd061f8f blocksize=2048 blocks=610 usedblocks=6 freeblocks=0 behindblocks=256 numfiles=1 numdirs=1 udfrev=1.02 udfwriterev=1.02 lastblock=610 integrity=closed accesstype=readonly softwriteprotect=no hardwriteprotect=no start=16, blocks=3, type=VRS start=32, blocks=16, type=MVDS start=64, blocks=1, type=LVID start=128, blocks=16, type=RVDS start=256, blocks=1, type=ANCHOR start=320, blocks=6, type=PSPACE start=353, blocks=1, type=ANCHOR ```

 

 

UDF multisession example (broken)

Loading session 1 and 2, so this appears as a multisession disc.

Most information by udisksctl is completly wrong (except the size).

blkid is wrong in an interesting way, as it shows the information of the first session. (I labeled the first session UDF-MulSess_1 and second session UDF-MulSess_2 to make them easily distinguishable) This looks like the related bug for blkid: https://github.com/karelzak/util-linux/issues/1161

udfinfo is correct, as it correctly shows the second sessions label UDF-MulSess_2 and other details of the second session.

udevadm and lsblk seem OK. But they just doesn't report the information essential to me (the filesystem).

multisession dump (broken) ``` $ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc $ udisksctl info -b /dev/sr1 /org/freedesktop/UDisks2/block_devices/sr1: org.freedesktop.UDisks2.Block: Configuration: [] CryptoBackingDevice: '/' Device: /dev/sr1 DeviceNumber: 2817 Drive: '/org/freedesktop/UDisks2/drives/CDEmu_Virt_2e_CD_2fDVD_ROM' HintAuto: true HintIconName: HintIgnore: false HintName: HintPartitionable: false HintSymbolicIconName: HintSystem: false Id: IdLabel: IdType: IdUUID: IdUsage: IdVersion: MDRaid: '/' MDRaidMember: '/' PreferredDevice: /dev/sr1 ReadOnly: true Size: 25845760 Symlinks: /dev/disk/by-path/platform-vhba-scsi-0:0:1:0 UserspaceMountOptions: $ udevadm info /dev/sr1 P: /devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1 N: sr1 S: disk/by-path/platform-vhba-scsi-0:0:1:0 E: DEVLINKS=/dev/disk/by-path/platform-vhba-scsi-0:0:1:0 E: DEVNAME=/dev/sr1 E: DEVPATH=/devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1 E: DEVTYPE=disk E: ID_BUS=scsi E: ID_CDROM=1 E: ID_CDROM_CD=1 E: ID_CDROM_CD_R=1 E: ID_CDROM_CD_RW=1 E: ID_CDROM_DVD=1 E: ID_CDROM_DVD_PLUS_R=1 E: ID_CDROM_DVD_R=1 E: ID_CDROM_DVD_RAM=1 E: ID_CDROM_MEDIA=1 E: ID_CDROM_MEDIA_CD=1 E: ID_CDROM_MEDIA_SESSION_COUNT=2 E: ID_CDROM_MEDIA_SESSION_LAST_OFFSET=24596480 E: ID_CDROM_MEDIA_TRACK_COUNT=2 E: ID_CDROM_MEDIA_TRACK_COUNT_DATA=2 E: ID_CDROM_MRW=1 E: ID_CDROM_MRW_W=1 E: ID_FOR_SEAT=block-platform-vhba-scsi-0_0_1_0 E: ID_MODEL=Virt._CD_DVD-ROM E: ID_MODEL_ENC=Virt.\x20CD\x2fDVD-ROM E: ID_PATH=platform-vhba-scsi-0:0:1:0 E: ID_PATH_TAG=platform-vhba-scsi-0_0_1_0 E: ID_REVISION=1.10 E: ID_SCSI=1 E: ID_SCSI_INQUIRY=1 E: ID_TYPE=cd E: ID_VENDOR=CDEmu E: ID_VENDOR_ENC=CDEmu\x20\x20\x20 E: MAJOR=11 E: MINOR=1 E: SCSI_MODEL=Virt._CD/DVD-ROM E: SCSI_MODEL_ENC=Virt.\x20CD/DVD-ROM E: SCSI_REVISION=1.10 E: SCSI_TPGS=0 E: SCSI_TYPE=cd/dvd E: SCSI_VENDOR=CDEmu E: SCSI_VENDOR_ENC=CDEmu\x20\x20\x20 E: SUBSYSTEM=block E: SYSTEMD_MOUNT_DEVICE_BOUND=1 E: TAGS=:uaccess:seat:systemd: E: USEC_INITIALIZED=76936869 $ blkid /dev/sr1 /dev/sr1: UUID="2010112200000000" LABEL="UDF-MulSess_1" TYPE="udf" $ lsblk /dev/sr1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr1 11:1 1 24,7M 0 rom $ ~/opt/udftools/bin/udfinfo /dev/sr1 udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second filename=/dev/sr1 label=UDF-MulSess_2 uuid=2010112200000000 lvid=UDF-MulSess_2 vid=UDF-MulSess_2 vsid= fsid=UDF-MulSess_2 fullvsid=20101122 owner= organization= contact= appid= impid=*UDF Image Creator winserialnum=0xbd08075b blocksize=2048 blocks=12620 usedblocks=12018 freeblocks=0 behindblocks=256 numfiles=2 numdirs=1 udfrev=1.02 udfwriterev=1.02 startblock=12010 lastblock=12620 integrity=closed accesstype=readonly softwriteprotect=no hardwriteprotect=no start=320, blocks=12018, type=PSPACE start=12026, blocks=3, type=VRS start=12042, blocks=16, type=MVDS start=12074, blocks=1, type=LVID start=12138, blocks=16, type=RVDS start=12363, blocks=1, type=ANCHOR ```
md-work commented 3 years ago

Maybe related: https://github.com/systemd/systemd/pull/17780

pali commented 3 years ago

I fixed both ISO and UDF superblock modules in libblkid to read last session from multisession optical disc:

But new version of util-linux with these fixes were not released yet. Note that without fixes blkid is not able to read multisession UDF disc. For testing you could try to compile util-linux from master git branch, blkid should properly detects multisession ISO and UDF discs.

I have also fixed udfinfo and udflabel in udftools 2.3 to properly handle multisession optical discs. You need to update udftools to version 2.3.

pali commented 3 years ago
$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc
...
$ ~/opt/udftools/bin/udfinfo /dev/sr1 
udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

Either UDF disc is corrupted or this looks like a bug in udfinfo. First Anchor Volume Descriptor Pointer should be available.

@md-work: Could you please provide me image of the /dev/sr1? E.g. generated by dd if=/dev/sr1 of=image.udf && gzip -9 image.udf.

By the way, udfinfo is very smart and can read Anchor Volume Descriptor Pointer (superblock of UDF) also from backup location, but this does not have to be supported by other UDF implementations (kernel, blkid, ...).

filename=/dev/sr1
label=UDF-MulSess_2
uuid=2010112200000000
lvid=UDF-MulSess_2
vid=UDF-MulSess_2
vsid=
fsid=UDF-MulSess_2
fullvsid=20101122
md-work commented 3 years ago
$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc
...
$ ~/opt/udftools/bin/udfinfo /dev/sr1 
udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

Either UDF disc is corrupted or this looks like a bug in udfinfo. First Anchor Volume Descriptor Pointer should be available.

I get the same "udfinfo" warning for a real disc without using cdemu. But the disc is being mounted without problems by the Linux kernel.

 

@md-work: Could you please provide me image of the /dev/sr1? E.g. generated by dd if=/dev/sr1 of=image.udf && gzip -9 image.udf.

Are you aware, that "dd" will only read the first session only!? This is why I used "cdrdao" before (it can properly image all sessions).

Also "dd" will exit with code 1 at the end of the block device. So gzip will never be executed.

If you're still sure that you want dd if=/dev/sr0 of=image.udf && gzip -9 image.udf I can gladly do that.

In case, I guess it shall be a real disc and not something loaded into cdemu!? I don't have the MINI_-_UDF-MulSess original disc anymore (just the image). But I have another real disc. It's the one I just tested "udfinfo" with (see above section).

 

By the way, udfinfo is very smart and can read Anchor Volume Descriptor Pointer (superblock of UDF) also from backup location, but this does not have to be supported by other UDF implementations (kernel, blkid, ...).

It would be nice if udisk would behave mostly like the Linux kernel. Or at least, if Linux can mount a disc udisks should be able to read the disc too.

pali commented 3 years ago

I get the same "udfinfo" warning for a real disc without using cdemu. But the disc is being mounted without problems by the Linux kernel.

That is why I wan to see that image as it can be a bug in udfinfo.

Are you aware, that "dd" will only read the first session only!?

dd read all data sessions (concatenated) with correct block numbering. It only does not provide ToC (information on which block first session ends and on which second session starts) and non-data sessions (e.g. audio), but this is not not our case.

Also "dd" will exit with code 1 at the end of the block device. So gzip will never be executed.

If you're still sure that you want dd if=/dev/sr0 of=image.udf && gzip -9 image.udf I can gladly do that.

So dd if=/dev/sr0 of=image.udf; gzip -9 image.udf then.

It would be nice if udisk would behave mostly like the Linux kernel. Or at least, if Linux can mount a disc udisks should be able to read the disc too.

This is feature request for udisks, I will let it for udisks developers.

md-work commented 3 years ago

Are you aware, that "dd" will only read the first session only!?

dd read all data sessions (concatenated) with correct block numbering. It only does not provide ToC (information on which block first session ends and on which second session starts) and non-data sessions (e.g. audio), but this is not not our case.

Here is the requested image (it's the disc I tested udfinfo with in my last comment): image_11MB_2-sessions.udf.gz

But as far as I can see it only contains files from the first session. On session two there are files over 11 MB in size. But the uncompressed dd image is only less than 2 MB in size.

$ dd if=/dev/sr0 of=image.udf; gzip -9 image.udf
dd: error reading '/dev/sr0': Input/output error
2432+0 records in
2432+0 records out
1245184 bytes (1.2 MB, 1.2 MiB) copied, 5.45237 s, 228 kB/s
$ lsblk /dev/sr0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0   11:0    1  35M  0 rom  
$ mount -o ro /dev/sr0 /mnt
$ ls -lhs /mnt/
total 11M
5.2M -r--r--r-- 1 nobody nobody 5.2M Jan 11 17:58 cdbxp_setup_4.5.8.7128_minimal_-_session_2_only.exe
5.2M -r--r--r-- 1 nobody nobody 5.2M Jan 11 20:22 isobuster_install_-_4.7isobuster_install_-_4.7_-_session_2_only.exe
2.0K -r--r--r-- 1 nobody nobody   34 Jan 11 21:09 session_1_and_2.txt
2.0K -r--r--r-- 1 nobody nobody   33 Jan 11 19:14 session_2_only.txt

Here's the cdrdao image from the same physical disc with full 11 MB of size for comparison: image_11MB2-sessions-_cdrdao.zip

pali commented 3 years ago

But as far as I can see it only contains files from the first session.

Looks like that your cdrom emulation software creates holes in /dev/sr0 between sessions and when trying to read these holes that cdrom emulation software returns -EIO (Input/output error) instead of returning zeros, which makes dd to immetiately stop.

Here's the cdrdao image from the same physical disc with full 11 MB of size for comparison: image_11MB2-sessions-_cdrdao.zip

You have put sessions into separate files. Such files are unusable for any UDF software as UDF filesystem is addressed by absolute block positions. So I need one continous file like it can be accessible by opening and reading /dev/sr0.

Could you provide just one data image file (either by cdrdao or any other software)? I do not need metadata (like ToC). I think that conv=noerror option for dd could instruct dd to skip errors. Or there is also ddrescue.

md-work commented 3 years ago

Looks like that your cdrom emulation software creates holes in /dev/sr0 between sessions and when trying to read these holes that cdrom emulation software returns -EIO (Input/output error) instead of returning zeros, which makes dd to immetiately stop.

I didn't use a cdrom software for that image. It's directly from a physical disc!

 

[...] You have put sessions into separate files. Such files are unusable for any UDF software as UDF filesystem is addressed by absolute block positions. So I need one continous file like it can be accessible by opening and reading /dev/sr0.

cdrdao can't do multiple sessions in one file.

Could you provide just one data image file (either by cdrdao or any other software)? I do not need metadata (like ToC). I think that conv=noerror option for dd could instruct dd to skip errors. Or there is also ddrescue.

I created another dd image with conv=noerror and it's now 35 MB in size. image-noerror_11MB_2-sessions.udf.gz I put a file with the content This file is on session 2 only on the second session and that string can indeed be found in that image (for example using strings command).

Nevertheless, Linux always mounts the first session only. Even when specifying mount -o session=2 .... I guess Linux needs that explicit errors to know there's a second session. But the noerror option converts that errors into zeros, so Linux doesn't get the clue for the second session.

 

Actually I think you might be on the wrong track! This is about sessions which is a feature outside the udf filesystem. You might have a look into the code udfinfo is using to detect the data on the second session.

pali commented 3 years ago

I didn't use a cdrom software for that image. It's directly from a physical disc!

Ok! Also cdrom drive is free to return -EIO for some block of data.

cdrdao can't do multiple sessions in one file.

:-( hard to use this sw for debugging then.

I created another dd image with conv=noerror and it's now 35 MB in size. image-noerror_11MB_2-sessions.udf.gz

Thank you! I will look at it and investiagate it.

Nevertheless, Linux always mounts the first session only. Even when specifying mount -o session=2 ....

Actually I think you might be on the wrong track! This is about sessions which is a feature outside the udf filesystem. You might have a look into the code udfinfo is using to detect the data on the second session.

I do not think so. Session borders are invisible in block device. block device does not see them. If you want to get information on which block is starting which session you need to use out-of-band data. For cd rom driver (/dev/sr0) you need to retrieve it e.g. via CDROMMULTISESSION ioctl. And both kernel and udfinfo is doing it. Kernel's udf driver by default read last session, but it only works when mounting /dev/sr0 (which provides CDROMMULTISESSION ioctl).

pali commented 3 years ago

Could you provide also output from /usr/bin/cd_sessions /dev/sr0 for that physical cdrom image?

pali commented 3 years ago

I created another dd image with conv=noerror and it's now 35 MB in size. image-noerror_11MB_2-sessions.udf.gz

This image is unusable... it looks like that dd omitted empty blocks instead of filling zeros. Could you try to use other tool? e.g. ddrescue?

md-work commented 3 years ago

Could you provide also output from /usr/bin/cd_sessions /dev/sr0 for that physical cdrom image? Used: http://www.13thmonkey.org/udfclient/releases/UDFclient.0.8.11.tgz


cd_sessions /dev/sr0 
Opening device /dev/sr0
Device has MMC profile 0x09
Disc info for disc in device /dev/sr0
MMC profile        : CD-R recordable
sequential         : yes
recordable         : yes
erasable           :  no
blankable          :  no
formattable        :  no
rewritable         :  no
mount raineer      :  no
packet writing     :  no
strict overwrite   :  no
blocking number    : 1
disc state         : incomplete (appendable)
last session state : empty
sectorsize         : 2048
Number of sessions     3
Session 0
start  at         0
ends   at         608
length for        608
next writable at  0
free blocks       0
packet size       0
    Session 1
            start  at         12010
            ends   at         17898
            length for        5888
            next writable at  0
            free blocks       0
            packet size       0

    Session 2
            start  at         24800
            ends   at         359845
            length for        335045
            next writable at  24800
            free blocks       335045
            packet size       0

Disc access statistics sector reads 0 (0 Kbyte) sector written 0 (0 Kbyte) switches 0



Interesting, although I added files to the disc just once there's a third "virtual!?" session!
I created another cdrdao image and also extracted the 3rd session this time:
[image_11MB_2plus1-sessions_-_cdrdao.zip](https://github.com/storaged-project/udisks/files/6029997/image_11MB_2plus1-sessions_-_cdrdao.zip)

cdrdao even didn't complain when trying to read a 4th and 5th session. But the resulting *.bin files where identical with the 3rd session. So I guess those 4th and 5th sessions don't really exist.

 

 

> > I created another dd image with conv=noerror and it's now 35 MB in size.
> > [image-noerror_11MB_2-sessions.udf.gz](https://github.com/storaged-project/udisks/files/6029141/image-noerror_11MB_2-sessions.udf.gz)
> 
> This image is unusable... it looks like that `dd` omitted empty blocks instead of filling zeros. Could you try to use other tool? e.g. `ddrescue`?

I'm not sure how to achieve this.
Can you give me a hint to follow?

Couldn't you further investigate this by loading the cdrdao image I provided into cdemu?
On my machine this triggers exactly the same problem as the physical disc does!

If you need a single image, you could use dd or ddrescue on the cdemu drive.

 

Alternatively please try creating a physical disc for yourself.
I put a very detailed explanation on how I created that physical disc in the first place beside the first images I posted here.
[udisks_udf_multisession_support.zip](https://github.com/storaged-project/udisks/files/5820318/udisks_udf_multisession_support.zip) -> `Multisession and UDF (CD and DVD).txt`
Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

You can get the cd burning software (CDBurnerXP) setup here:
https://cdburnerxp.se/downloadsetup.exe
https://www.heise.de/download/product/cdburnerxp-35927
pali commented 3 years ago

Interesting, although I added files to the disc just once there's a third "virtual!?" session!

I think this is because you have not closed disc yet, so this third session is a place where new session can start. See next writable at.

I'm not sure how to achieve this. Can you give me a hint to follow?

I think that simple call /usr/bin/ddrescue /dev/sr0 image.udf should be enough. Other switches are described in manual page.

Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

And this is the issue, I do not have any native for a long time.

md-work commented 3 years ago

I'm not sure how to achieve this. Can you give me a hint to follow?

I think that simple call /usr/bin/ddrescue /dev/sr0 image.udf should be enough. Other switches are described in manual page.

image-ddrescue_11MB_2-sessions.udf.gz

$ ddrescue /dev/sr0 image-ddrescue_11MB_2-sessions.udf
GNU ddrescue 1.23
Press Ctrl-C to interrupt
     ipos:   24300 kB, non-trimmed:        0 B,  current rate:       0 B/s
     opos:   24300 kB, non-scraped:        0 B,  average rate:    197 kB/s
non-tried:        0 B,  bad-sector:    49152 B,    error rate:     512 B/s
  rescued:   36605 kB,   bad areas:        3,        run time:      3m  4s
pct rescued:   99.86%, read errors:      100,  remaining time:         n/a
                              time since last successful read:      1m 21s
Finished

Also tried dd_rescue (differs from ddrescue). But the output file was exactly the same.

 

Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

And this is the issue, I do not have any native for a long time.

I confess, I used my girlfriends notebook for that part because I don't have any native Windows too :-D

pali commented 3 years ago

Perfect, output from cd_sessions and file image-ddrescue_11MB_2-sessions.udf.gz seems to be usable! I will investigate it and let you know.

What I have learnt today? Do not use dd but rather ddrescue.

md-work commented 3 years ago

Definitely learned that too. I though you need a session aware software to get all bits from a disc, but ddrescue seems to do the job too.

Nevertheless, I can't reproduce the exact udisks behavior of the physical disc when using the ddrescue image. I tried the ddrescue image via cdemu, losetup and by passing it to a qemu vm. In all cases udisks behaved like the disc had just the first session.

Only by using the cdrdao image via cdemu I can reproduce the udisks problem in the same manner as with the physical disc! I guess this is because cdemu is able to provide suitable out-of-band-data.

pali commented 3 years ago

Nevertheless, I can't reproduce the exact udisks behavior of the physical disc when using the ddrescue image.

That is expected. Image itself does not contain session information. This is stored only in out-of-band data. But you have provided it in cd_sessions output, so I can "simulate" it (udftools have --startblock and --lastblock options).

pali commented 3 years ago

udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

And now I see where is the issue.

Lets read first session:

$ ./udfinfo/udfinfo -b 2048 --startblock=0 --lastblock=608 image-ddrescue_11MB_2-sessions.udf
filename=image-ddrescue_11MB_2-sessions.udf
label=UDF-MulSess_1
uuid=2010112200000000
lvid=UDF-MulSess_1
vid=UDF-MulSess_1
vsid=
fsid=UDF-MulSess_1
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd061f8f
blocksize=2048
blocks=17898
usedblocks=9
freeblocks=0
behindblocks=17545
numfiles=2
numdirs=1
udfrev=1.02
udfwriterev=1.02
lastblock=608
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=16, blocks=3, type=VRS
start=32, blocks=16, type=MVDS
start=64, blocks=1, type=LVID
start=128, blocks=16, type=RVDS
start=256, blocks=1, type=ANCHOR
start=320, blocks=9, type=PSPACE
start=352, blocks=1, type=ANCHOR

No issue, it is correct.

And now second:

$ ./udfinfo/udfinfo -b 2048 --startblock=12010 --lastblock=17898 image-ddrescue_11MB_2-sessions.udf
udfinfo: Warning: First Anchor Volume Descriptor Pointer not found, using second
udfinfo: Warning: Partition Space overlaps with other blocks
filename=image-ddrescue_11MB_2-sessions.udf
label=UDF-MulSess_2
uuid=2010112200000000
lvid=UDF-MulSess_2
vid=UDF-MulSess_2
vsid=
fsid=UDF-MulSess_2
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd08075b
blocksize=2048
blocks=17898
usedblocks=17308
freeblocks=0
behindblocks=0
numfiles=4
numdirs=1
udfrev=1.02
udfwriterev=1.02
startblock=12010
lastblock=17898
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=320, blocks=17308, type=PSPACE
start=12026, blocks=3, type=VRS
start=12042, blocks=16, type=MVDS
start=12074, blocks=1, type=LVID
start=12138, blocks=16, type=RVDS
start=17641, blocks=1, type=ANCHOR
start=17897, blocks=1, type=ANCHOR

Message udfinfo: Warning: Partition Space overlaps with other blocks can be ignored as on multisession disk data space may continue from one session to another or for space effectivity may overlap with other data (if it is properly created with holes).

But message udfinfo: Warning: First Anchor Volume Descriptor Pointer not found, using second is big warning as the first AVDP is entry point for UDF. Without first AVDP UDF filesystem is unusable. udfinfo is slightly smarter than other UDF implementations and can read AVDP from backup locations (and we are happy that there are two backups at the end of disk).

First AVDP must be at fixed block start+256. So at block 12266 for second session. Lets look what is stored at this block (via udfdump -S):

00012266    12266       TAG: descriptor 2, serial_num 1 at sector 256, crc length 496 bytes
                Anchor
                        Main    volume descriptor set at 12042 for 32768 bytes
                        Reserve volume descriptor set at 12138 for 32768 bytes

So AVDP is at block 12266, but has stored information in its data that is placed on block 256 (instead of 12266). 12266 != 256 meaning that content of block 12266 is broken and therefore ignored.

This seems to be a bug of software which generated second session of UDF filesystem.

On the other hand on block 17641 is stored backup of AVDP and it is correct (17641 = 17641):

00017641    17641       TAG: descriptor 2, serial_num 1 at sector 17641, crc length 496 bytes
                Anchor
                        Main    volume descriptor set at 12042 for 32768 bytes
                        Reserve volume descriptor set at 12138 for 32768 bytes

(so udfinfo used it)

pali commented 3 years ago

So I'm not surprised now that this udf multisession cd image cannot be detected, it is broken in a way that it is not valid udf image.

pali commented 3 years ago

Anyway, seems that correct way to call dd is with arguments conv=noerror,sync (noerror is not enough, that sync should add required zero bytes). Can you just recheck if dd conv=noerror,sync produce same output as dddrescue?

EDIT: seems that this is still not enough. According to https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html additional iflag=fullblock is needed. So: dd conv=noerror,sync iflag=fullblock if=/dev/sr0 of=image.udf

md-work commented 3 years ago

Anyway, seems that correct way to call dd is with arguments conv=noerror,sync (noerror is not enough, that sync should add required zero bytes). Can you just recheck if dd conv=noerror,sync produce same output as dddrescue?

EDIT: seems that this is still not enough. According to https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html additional iflag=fullblock is needed. So: dd conv=noerror,sync iflag=fullblock if=/dev/sr0 of=image.udf

Yes, it's identical with the ddrescue image.

 

[...] This seems to be a bug of software which generated second session of UDF filesystem.

Bad thing... Nevertheless, because most systems (including the Linux-Kernel and Windows - both tested) can handle this I guess this is a maybe common bug.

 

I finally found the physical disc I read the Test-Media-Mini image from. (see cdrdao image in udisks_udf_multisession_support.zip) Here's an image read via ddrescue. Test-Media-Mini_ddrescue.udf.gz

For this Test-Media-Mini a recent blkid.static compiled from master branch actually reports the correct label of the second session. https://github.com/karelzak/util-linux/commit/17d5b264367debb745b678fcafacbaa938b29455 (compiled from this commit) blkid-2.23.1 and blkid-2.23.2 still report the incorrect first session label.

pali commented 3 years ago

Yes, it's identical with the ddrescue image.

Ok, so we should always call dd with arguments conv=noerror,sync iflag=fullblock.

Test-Media-Mini_ddrescue.udf.gz

This image has same issue. And also has impid=*UDF Image Creator, so seems that this UDF Image Creator is broken...

HeroesLoveToRoujiamo commented 3 years ago

Regarding the multisession UDF, the kernel also needs to be fixed. This is my code submission, and it should not have been released yet:https://lore.kernel.org/lkml/20210114075741.30448-1-changlianzhi@uniontech.com/. Use with code updates related to the https://github.com/karelzak/util-linux/issues/1161 issue.@pali

pali commented 3 years ago

I guess that above kernel patch should be backported also to stable kernel releases

HeroesLoveToRoujiamo commented 3 years ago

This is beyond my control, I actually think so too

pali commented 3 years ago

When sending such bugfix patch, just add Cc: stable@vger.kernel.org line into email body (commit message) and patch would be automatically backported (https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html). Also if you mention that patch is suitable for stable releases then maintainers would add that line...

HeroesLoveToRoujiamo commented 3 years ago

Thank you very much, I learned another knowledge point