search

pali / udftools

Linux tools for UDF filesystems and DVD/CD-R(W) drives
GNU General Public License v2.0
106 stars 30 forks source link

Quite dramatic failures with cdrwtool #33

Closed scdbackup closed 3 years ago

scdbackup commented 5 years ago

Hi,

pali wrote in issue 32:

Also I asked on some Debian mailing list if pktsetup should be moved from udftools package and conclusion was that it should say where is.

Package shuffling causes inconvenience to old users. Sometimes i advise the use of /dev/pktcdvdN with formatted DVD-RW, if they shall behave much like DVD-RAM, DVD+RW, BD-RE. It is convenient to know where Debian et.al. have it.

I think, cdrwtool is partly off UDF topic, too. It combines hardware formatting of CD-RW with on-topic UDF formatting.

Anyways, there is some bug with it. I experienced it in april this year:

$ cdrwtool -d /dev/sr0 -q ... wait_cmd: Bad address Command failed: bb 00 ff ff 08 40 00 00 00 00 00 00 - sense 00.00.00 set speed

BB is SET SPEED. Sense 0,0,0 looks like not the drive but the kernel raised protest. (One would have to print SG_IO diagnostics like struct sg_io_hdr_t.host_status and sg_io_hdr_t.driver_status. But cdrwtool.c still uses elderly CDROM_SEND_PACKET instead of SG_IO.)

The failure is known since long time

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511059 "udftools: cdrwtool -d /dev/hdc -q fails"

But right now, it fails differently here:

$ cdrwtool -d /dev/sr4 -q ... Settings for /dev/sr4: Fixed packets, size 32 Mode-2 disc ... Initiating quick disc blank Disc capacity is 295264 blocks (590528KB/576MB) Formatting track ... Writing UDF structures to disc Quick setup complete! $

But then, /dev/sr4 is gone and the kernel simply reports

Aug 27 11:01:50 ts6 kernel: [...] usb 1-1.3: USB disconnect, device number 24

To my luck it is a USB drive. A power cycle brought it back. Inspecting the CD-RW by libburn yields that it is not formatted. As final proof i filled it with 700 MiB of data, which a formatted CD-RW could not take all.

So again, there seems to be a problem on kernel level. (A formatting experiment with libburn on CD-RW is on my todo list. Currently i apply formatting only to DVD and BD media.)

Have a nice day :)

Thomas

pali commented 5 years ago

So again, there seems to be a problem on kernel level.

So then please report bug to kernel.

scdbackup commented 5 years ago

Hi,

even if i'd find somebody in the kernel community who would be interested, then the first reply would be about the use of long deprecated ioctl(CDROM_SEND_PACKET) instead of ioctl(SG_IO).

By the theory of "kernel level" i mean the use of the SCSI transport facility and maybe of some CDROM_* ioctls. Our main chance as userlanders is to find out which gestures trigger the problem and to then avoid them.

First opportunity to learn more: Does it work for you ?

cdrwtool -d /dev/sr0 -q

Have a nice day :)

Thomas

pali commented 5 years ago

Currently I do not have there optical mechanic. But at time of releasing udftools 2.1 I tested it and worked fine...

scdbackup commented 5 years ago

Hi,

after next release of libburn (hopefully soon) i will teach it CD-RW formatting by the same command 04 FORMAT UNIT as in cdrwtool.c. This will show whether my kernel or drives react allergic to this central gesture of cdrwtools.c or rather to some of its surrounding activities. libburn works nearly only by SG_IO. So we can hopefully exclude some of the suspects in cdrwtools.c, depending on the outcome of the experiment.

Have a nice day :)

Thomas

scdbackup commented 5 years ago

Hi,

i tested cdrwtool as of Debian stable via a Debian Live 10 ISO.

With the built-in SATA-attached drive it worked fine. Slowly, but obviously with success. The CD-RW appears to libburn as in closed state with one track of the size which cdrwtools had announced: 295264 blocks.

Then i took the USB drive which suffered the sudden kernel disconnect after a cdrwtool run on my workstation and attached it to the test machine with the Debian Live system running (kernel 4.17). The run of cdrwtool ended much quicker than with the SATA-attached drive. No error indication. But the device file of the drive was gone like on my workstation. The CD-RW is perceived by libburn as blank with 703 MiB capacity.

The USB drive is a halfways modern Blu-ray burner: Drive type : vendor 'ASUS' product 'BW-16D1HT' revision '1.01' The CD-RWs are quite old ones: Media product: 97m26s65f/79m59s74f , CMC Magnetics Corporation

It would be interesting to learn what shocks drive, USB-SATA box, or kernel so much that the virtual plug pops out of the socket. Also, the UDF formatting messages did not indicate any problem. So how could write operations succeed when the device was being disconnected ?

Have a nice day :)

Thomas

pali commented 5 years ago

So how could write operations succeed when the device was being disconnected ?

Definitely a bug. But looks like in kernel or at HW level.

callegar commented 4 years ago

For me cdrwtool seems to work on an (older) USB DVD drive, but fails on a (modern) USB Blu-Ray/DVD drive.

Any idea of what this may mean:

cdrwtool -d /dev/sr1 -q
using device /dev/sr1
1178KB internal buffer
setting write speed to 12x
wait_cmd: Input/output error
Command failed: 55 10 00 00 00 00 00 00 3c 00 00 00 - sense 05.26.00
00 00 00 00 00 00 00 00 05 32 20 27 0a 00 00 00 20 00 00 00 00 20 00 96 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08 00 
mode_select: Input/output error
set_write
can't unlock door

On the older device I get:

cdrwtool -d /dev/sr1 -q
using device /dev/sr1
1088KB internal buffer
setting write speed to 12x
Settings for /dev/sr1:
        Fixed packets, size 32
        Mode-2 disc

I'm going to do a quick setup of /dev/sr1. The disc is going to be blanked and formatted with one big track. All data on the device will be lost!! Press CTRL-C to cancel now.
ENTER to continue.

Initiating quick disc blank
Disc capacity is 295264 blocks (590528KB/576MB)
Formatting track
callegar commented 4 years ago

On the Blu-ray drive also cd-info seems to fail: ++ WARN: error in ioctl CDROMREADTOCHDR: Input/output error kernel error? broken drive firmware?

scdbackup commented 4 years ago

Hi,

sense 05.26.00

If this is SCSI KEY.ASC.ASCQ then it means:

"Illegal request. Invalid field in parameter list."

The failed command was obviously 0x55 MODE SELECT, sending 60 bytes of mode page data. Assuming that the shown 60 hex bytes are those data, it was a mode page 05 "Write Parameters" with:

LS_V = 1 (Link Size field is Valid) Write Type = 0 (Packet), FP = 1 (Fixed Packet size) Track Mode = 7 (Control nibble in Q Sub-channel, data track, increment) Data Block Type = 10 (Mode 2, CD-ROM XA, form 1) Link Size = 0 (seems ok because the prescribed error is 5.64.00) Session Format = 32 (CD-ROM XA) Packet Size = 32 (XA blocks of 2048 bytes payload) Audio Pause Length = 150 (is the default value)

Now we can guess which of these parameters appears illegal to the drive.

++ WARN: error in ioctl CDROMREADTOCHDR: Input/output error

This is probably from sr_read_tochdr() in Linux drivers/scsi/sr_ioctl.c It performs a SCSI command READ TOC/PMA/ATIP with: MSF = 0 (i.e. addresses shall be expressed as LBA) Format = 0 (Formatted TOC) Track/Session Number = 0 (with Format 0 it is the track number) Allocation Length = 12 (that's strange, 4 would suffice for the purpose)

By requesting 12 bytes of reply, Linux assumes the existence of a first track entry. The drive is allowed to return response data which "consist of four header bytes and zero or more track descriptors" (MMC-5 6.26.3.2.1). The lower level of Linux SG_IO is known to hate drive replies which deliver less bytes than requested.

Does this happen with a CD which bears readable content ?

I wonder what drive this is (vendor, model) and how it copes with burn jobs on CD, DVD, and BD by burn programs like K3B, Xfburn, xorriso, ...

Have a nice day :)

Thomas

callegar commented 4 years ago

Drive is branded as Maxell, but actually reports being a Pioneer BD-RW BDR-TD05 at release 1.00. I have now seen that Sony has a 1.01 firmware upgrade for this drive and looking at the internet there appears to be also an 1.02, yet with unknown origin. Unclear to me if I should try an upgrade. The drive appears to work fine with K3B on write once media, in burning operation, but fails burn+verify saying that it cannot find track and to insert medium. With erasable medium it fails to erase, again saying "load" all the time. Thanks for your help!

scdbackup commented 4 years ago

Hi,

your experience with K3B does not give much trust in the drive.

We should try to find out whether it works for the use cases other than CD-RW formatting. If it is generally weird, then problems with cdrwtool are not likely to be software-related.

So if you have a few media left for exercising, then contact me on mailing list bug-xorriso@gnu.org or in private via scdbackup@gmx.net . Tell what media you have and i will propose xorriso runs which will challenge the drive's capabilities.

For determining medium type and state: [code] xorriso -outdev /dev/sr0 -toc [/code]

Have a nice day :)

Thomas

callegar commented 4 years ago

More checks: seems happy either with cd-info and xorriso as long as I have a CD with readable content.

callegar commented 4 years ago

And xorriso is happy anyway (returns "blank" with blank CD).

Final note: firmware upgrade from sony does not install (says no target found, probably expecting a pioneer or sony model string somewhere, while my device is branded MAXELL MBR-6U).

scdbackup commented 4 years ago

Hi,

seems happy either with cd-info and xorriso as long as I have a CD with readable content.

So you possibly found a kernel bug around CDROMREADTOCHDR on blank CD. (Before accusing the kernel of doing it wrong, one would have to find out the drive's reply to the SCSI command and whether it would work better with requested data length 4 rather than 12.)

I guess it happens only if the table-of-content of the CD is empty. So it makes not much difference in practice whether the ioctl fails or tells that there are no tracks. The problem seems to be very old. I can reproduce it on kernel 3.16.

Nevertheless, the on-topic riddle here is why the drive rejects the mode page with its very cdrwtools-specific values. My best guess is the combination of CD-ROM XA and packet writing. But i can be wrong.

Have a nice day :)

Thomas

scdbackup commented 4 years ago

Hi,

i dug in the cdrwtool/ files about the code which decides for CD-ROM XA. It appears that the user can switch between Mode 2 CD-ROM XA and Mode 1 Data by options -w "mode2" and -w "mode1". Mode 2 is default.

So to test my unqualified guess, re-try your cdrwtool runs with additional option

-w mode1

Have a nice day :)

Thomas

pali commented 3 years ago

At the end of last year I tested cdrwtool on SATA DVD burner with CD-RW medium and it worked fine. I will try to find some time and test cdrwtool this month again but I'm afraid that I have only one DVD optical drive for testing, probably same which I used for testing last year, which worked fine.

pali commented 3 years ago

@callegar: Have you tried mode1 if something was changed?

pali commented 3 years ago

So you possibly found a kernel bug around CDROMREADTOCHDR on blank CD. (Before accusing the kernel of doing it wrong, one would have to find out the drive's reply to the SCSI command and whether it would work better with requested data length 4 rather than 12.)

I guess it happens only if the table-of-content of the CD is empty. So it makes not much difference in practice whether the ioctl fails or tells that there are no tracks. The problem seems to be very old. I can reproduce it on kernel 3.16.

It is still better if ioctl does not fail and correctly reports that TOC is empty.

callegar commented 3 years ago

Sorry I forgot to report. As a matter of fact I started using just the older drive and taking out the newer BD one just for Blueray.

Mode 1 makes no difference:

using device /dev/sr0
mode1
1178KB internal buffer
setting write speed to 12x
wait_cmd: Input/output error
Command failed: 55 10 00 00 00 00 00 00 3c 00 00 00 - sense 05.26.00
00 00 00 00 00 00 00 00 05 32 20 27 08 00 00 00 00 00 00 00 00 20 00 96 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
mode_select: Input/output error
set_write
can't unlock door
scdbackup commented 3 years ago

Hi,

So you possibly found a kernel bug around CDROMREADTOCHDR on blank CD.

It is still better if ioctl does not fail and correctly reports that TOC is empty.

I made kernel experiments on 5.10-rc3 (loitering on a new machine). Reducing the number of requested bytes for READ TOC/PMA/ATIP from 12 to 4 did not silence the I/O error of ioctl(CDROMREADTOCHDR).

With an ASUS DRW-24D5MT at SATA and a blank CD-RW, i get i/o error from sr_read_tochdr(), because in drivers/scsi/sr_ioctl.c function sr_do_ioctl() the call to scsi_execute() returns 0x08000002 and sense data sense_key=5, asc=0x24, ascq=0. These sense data mean "INVALID FIELD IN CDB". So the drive hates the request.

The same request is ok with a CD-RW which holds a data session.

The sense code 5,24,0 is not specific to the situation. It would become tricky to override the EIO in this case. sr_read_tochdr() should rather inquire the medium state, recognize that it is "blank", and behave in a way that is distinguishable from harder I/O problems. E.g. by setting .cdth_trk0=1 and .cdth_trk1=0 and returning success.

Problem is that the kernel currently hides the SCSI command for medium state inquiry in drivers/cdrom/cdrom.c function cdrom_get_disc_info(), which sr_ioctl.c cannot reach.

I have a patch to change that shortcomming as part of a fix for the wrong size of 2048 bytes which ioctl(BLKGETSIZE) attributes to blank media. But i have already two bug fixes dangling on linux-scsi mailing list since weeks. No replies. No attention. They don't even ignore me ...

So if there were some hope for getting a change of sr_read_tochdr() into the kernel, i could quite surely provide the code. But as it is kernel-community-wise, you will have to take EIO from ioctl(CDROMREADTOCHDR) as "maybe the medium is blank, maybe the drive is dead, maybe something completly different is wrong". It does not necessarily say that you cannot format the medium or write on it.

Sergio Callegari wrote:

Mode 1 makes no difference: Command failed: 55 10 00 00 00 00 00 00 3c 00 00 00 - sense 05.26.00 00 00 00 00 00 00 00 00 05 32 20 27 08 00 00 00 00 00 00 00 00 20 00 96 00 ... 00

Data Block Type was now 8 = Mode 1. Session Format was now 0 = "CD-DA, or CD-ROM or other data discs" There is still Fixed Packet Size and Packet Size 32.

The sense code is still "Illegal request. Invalid field in parameter list.".

So we learned not much, i fear.

Have a nice day :)

Thomas

pali commented 3 years ago

I have a patch to change that shortcomming as part of a fix for the wrong size of 2048 bytes which ioctl(BLKGETSIZE) attributes to blank media. But i have already two bug fixes dangling on linux-scsi mailing list since weeks. No replies. No attention. They don't even ignore me ...

So if there were some hope for getting a change of sr_read_tochdr() into the kernel, i could quite surely provide the code.

Add CC: pali at kernel (dot) org to the next email replies or for next patches.

scdbackup commented 3 years ago

Hi,

Add CC: pali at kernel (dot) org to the next email replies or for next patches.

Thank you for your support.

But in this special case i begin to get scruples about changing the behavior of the old ioctl(CDROMREADTOCHDR).

Currently EIO is the indication of a blank medium, if the drive takes offense. I would need to find at least one drive which does not take offense from READ TOC/PMA/ATIP on a blank CD-RW. If this drive yields a suitable result, then i could justify to "fix" all other drives to show the same behavior.

I'm now done with all of mine which are currently operational: ASUS DRW-24D5MT : EIO PIONEER BD-RW BDR-S09 : EIO LG BDDVDRW GGC-H20L : EIO LG BD-RE GGW-H20L : EIO Optiarc BD RW BD-5300S : EIO ASUS BW-16D1HT : EIO LG BD-RE BH16NS40 : EIO

So i cannot find any traditional template for a better behavior of the ioctl.

If the drive is empty or the tray is out, i get ENOMEDIUM. So as odd as the current behavior is, it is distinguishable from other regular situations, just not from failures of hardware.

Can we justify the wish for a change in behavior ?

Have a nice day :)

Thomas

callegar commented 3 years ago

If there is anything that I may try on my weird Maxell MBR-6U (but maybe Pioneer BD-RW BDR-TD05 as reported by K3B), I'd be glad to help. Incidentally and out of curiosity is there anything that even before having a fix accepted in the kernel can be done via a kernel module (e.g. exposing for a different/fixed kernel interface that usermode tools can be modified to use at least for testing purposes?)

scdbackup commented 3 years ago

Hi,

is there anything that even before having a fix accepted in the kernel can be done via a kernel module

Rather not.

The EIO thing is about a clumsy implementation of ioctl(CDROMREADTOCHDR) long years ago. (It belongs to the "new" uniform CDROM driver of 1999 https://www.kernel.org/doc/html/latest/cdrom/cdrom-standard.html ) It does not prevent the use of cdrwtool.

The problem with cdrwtool and the BD drive is not related to the kernel. cdrwtool sent the SCSI command for setting up the desired write mode and the drive responded with refusal. The command and its transmitted data look ok for the purpose of udftools.

(In libburn i follow a sequential model and would request write type TAO or SAO instead of Packet. So i have no experience with what drives are willing to do around Packet writing.)

Have a nice day :)

Thomas

wdlkmpx commented 3 years ago

I never used cdrwtool, but I guess there's a reason to get rid of that tool or improve it dramatically so that it becomes THE tool.

I have like 5 dvd drives and 1 bluray drive, but I don't use them anymore. Storage? External hard disk. Movies in a blu ray player? USB sticks.

I've had problems with advanced burning software on Windows and Linux, specially double-layer DVD's. Optical discs are really hard to work with. The best results are achieved with only 1 session.

I guess the ability to create a filesystem is enough, i.e. udf.iso, then there's the advanced dvd/bluray burning software to burn that ISO.

How's that?

pali commented 3 years ago

So seems that EIO is the way how kernel reports empty medium. So it is really better to not change behavior.

@scdbackup: Anyway, if you need help with existing or new kernel patches, add me to the copy.

@wdlkmpx: cdrwtools is needed only for formatting CD-RW medias to UDF and later use it for packet writing & mounting it in R/W mode via kernel drivers. For DVD and BD discs is cdrwtool not used.

mkudffs allows you to create UDF image (and with kernel help then fill it with content). xorriso allows you to create ISO image and fill it with files. And then you can use your favorite burning tool to write either UDF or ISO image to your optical media. This is fine and correct way.

Packet writing is there to allow usage of kernel infrastructure to modify optical discs without need for burning software. This is to "simulate" behavior of disks and usb flash drives, real R/W media: mount, copy/modify, umount.

scdbackup commented 3 years ago

Hi,

using the cdrwtools from git clone https://github.com/pali/udftools i fail to format a CD-RW on all three drives which are attached to the test machine:

ASUS DRW-24D5MT (2019) SATA PIONEER BD-RW BDR-S09 (2020) SATA HL-DT-ST BD-RE BH16NS40 (2013) USB-SATA bridge

ASUS and HL-DT-ST (LG) report what the modern drive of callgar reports:

Command failed: 55 10 00 00 00 00 00 00 3c 00 00 00 - sense 05.26.00

The Pioneer gets over this stage, but later fails with

Formatting track wait_cmd: Input/output error Command failed: 04 17 00 00 00 00 00 00 00 00 00 00 - sense 05.64.00

That's a FORMAT UNIT command. MMC-5 says about the hex 17:

FmtData is 1 Format Code is 7

FmtData is prescribed to be 1, but for Format Code the MMC-5 specs say:

"6.5.2.4 The Format Code identifies the parameter list format. The Format Code shall be set to one (001b)."

But ye olde MMC-1 prescribes

"For CD-RW, the Format Code shall be set to seven (111b)."

Further, the layout of the Format Descriptors differs between both Format Codes.

I modified cdrwtools by writing a new function format_disc_code1() which i called from quick_setup() instead of format_disc(). It follows the description of MMC-5 about Format Type 0x10 "CD-RW/DVD-RW Full Format":

FORMAT UNIT : 04 11 00 00 00 00 00 00 descriptors: 00 00 00 08 00 04 81 60 40 00 00 20

It succeeded, but the FORMAT UNIT command obviously ended before the drive was ready. The write commands for creating the UDF filesystem failed with

Command failed: 2a 00 00 00 00 00 00 00 20 00 00 00 - sense 02.04.04

where the sense code means

2 04 04 LOGICAL UNIT NOT READY, FORMAT IN PROGRESS

As one can see in above descriptor bytes, i did not set the Immed bit. (That would be bit 1 in byte 1 of the descriptor bytes.)

So i implemented wait_for_unit_ready() which issues TEST UNIT READY commands every 0.5 seconds and loops up to 20 minutes until either the reply is positive, or an error other than "NOT READY" is replied.

The Pioneer drive gets ready with a "12x" CD-RW about 500 seconds after the FORMAT UNIT command was executed. Then the program runs up to a final

Quick setup complete! can't unlock door

After due eject and reload, the drive can be mounted

mount /dev/pktcdvd/pktcdvd0 /mnt/iso

and mount reports

/dev/pktcdvd/pktcdvd0 on /mnt/iso type udf (rw,relatime,utf8)

(It's about time i turn to my kernel patches. I have a ioctl(CDROM_SIMUL_CHANGE) which simulates an eject-reload cycle. Surprisingly simple because such a feature is alreay in sr but not reachable by userland.)

I will try to shed light on the other two drives' refusal to set up the write parameters by command 55 MODE SELECT.

@pali:

My test code is quite raw. Especially i'd need to get access to write_params_t.packet_size in order to get MODE SELECT and FORMAT UNIT in sync about the fixed packet size. MMC-5 prescribes failure if the sizes in mode page 5 and format 0x10 descriptor differ.

Shall i elaborate my hack towards a solid fallback if the old MMC-1 format descriptor layout yields failure ? Or shall i rather try the modern layout first ?

(There is also a flaw in sig_progress() which expects its local struct request_sense to be used by wait_cmd(). But it uses an own local variable for catching the sense reply. I had to make it static for my hack.)

Have a nice day :)

Thomas

scdbackup commented 3 years ago

Hi,

by much forth and back i learned that the refusal on 55 MODE SELECT by my ASUS and LG burners is a bit volatile but that in general both drives accept the desired mode page 5 data if the CD-RW is closed and freshly inserted before cdrwtool is run.

Blank or appendable CD-RW caused failure, except in two not reproducible incidents yesterday. One of them saw failure with packet size 32 and then success with packet size 16. There was one incident when the LG failed on a closed CD-RW. This failure was reproducible until i ejected and loaded the medium.

@callegar: Can you confirm my findings ?

To bring a CD-RW surely into closed state, do for example:

dd if=/dev/zero bs=1M count=10 | \
xorriso -as cdrecord -v -eject dev=/dev/sr0 blank=as_needed -

If it does not get ejected after the burn run, eject manually. Then re-load and run for example

cdrwtool -d /dev/sr0 -q

It should get up to this prompt:

I'm going to do a quick setup of /dev/sr0. The disc is going to be blanked
and formatted with one big track. All data on the device will be lost!!
Press CTRL-C to cancel now.
ENTER to continue.

A CD-RW may be brought to blank state by

xorriso -as cdrecord -v -eject dev=/dev/sr0 blank=as_needed

This one would fail with cdrwtool on my ASUS and LG.

So far so good. With closed CD-RW the program gets to FORMAT UNIT on all three drives.

But the LG BD burner gets stuck with the old Format Code 7. No error reply. After about 10 minutes there was no blinking or audible activity of the drive any more. I gave it a power cycle after half an hour. With new Format Code 1 it did not work either. There is no early end of FORMAT UNIT, unlike than with the Pioneer drive.

So i set the Immed bit in the Format List Header. The FORMAT UNIT command ends after about 5 seconds and the TEST UNIT READY loop ends after about 530 seconds. Only visible problem is the usual final message

can't unlock door

Next i equipped the Code 7 function format_disc() with Immed and a call of my new wait_for_unit_ready(). This behaves like format_disc_code1(): Waiting loop starts after about 5 seconds and ends after about 530 seconds. Formatting and writing succeed.

So the problem with the LG burner is not about the Format Code but about the long running FORMAT UNIT command.

The ASUS DVD burner succeeded with unmodified cdrwtool. So it is not a kernel issue but an issue with the LG drive or its USB box (a DeLOCK 5.25 inch SATA-USB enclosure).

This raises the question why USE_IMMED is defined to 0.

Is it only because there is not wait_for_unit_ready() yet ? (Without it, Immed will cause error replies with the next SCSI commands.)

Or is it because some very old drives hate it ? (I use Immed in libburn's BLANK and FORMAT UNIT commands unconditionally. My scope are MMC compliant drives. I.e. all drives from this century. I never got complaints which would point towards Immed being bad.)

@pali: It looks like boss decisions are needed how to arrange the workarounds. I can imagine two approaches which avoid the LG getting stuck and promise success on all three drives:

When a decision is made and implemented, we probably need tests with very old drives. Regrettably my oldest from 2003 is for IDE/ATAPI and none of my computers has an IDE controller. I dimly remember that it worked fine with BLANK and Immed bit.

Have a nice day :)

Thomas

pali commented 3 years ago

issues TEST UNIT READY commands every 0.5 seconds and loops up to 20 minutes until either the reply is positive, or an error other than "NOT READY" is replied.

Maybe you should use WAIT_BLANK macro (or derivate from it real time interval, it is 60 minutes) instead of 20 minutes. As currently format_disc() for GPCMD_FORMAT_UNIT command is using WAIT_BLANK timeout.

Or cannot you set USE_IMMED bit?

Shall i elaborate my hack towards a solid fallback if the old MMC-1 format descriptor layout yields failure ? Or shall i rather try the modern layout first ?

So problem is that old MMC-1 requires to use Format Code is 7 for CD-RW and new MMC-7 requires Format Code is 1. New drivers do not accept requirements from MMC-1 and old drivers obviously do not support MMC-7 (when was only MMC-1 present). Do we know other applications handles this issue? If yes, we should do the same thing.

If not, I'm for issuing new MMC-7 way with fallback to old MMC-1. On more places is both kernel and udftools trying to use new standards and if it fails then fallback to old.

pali commented 3 years ago

This raises the question why USE_IMMED is defined to 0.

Is it only because there is not wait_for_unit_ready() yet ? (Without it, Immed will cause error replies with the next SCSI commands.)

Or is it because some very old drives hate it ? (I use Immed in libburn's BLANK and FORMAT UNIT commands unconditionally. My scope are MMC compliant drives. I.e. all drives from this century. I never got complaints which would point towards Immed being bad.)

I have no idea. This is is old code, it was written 20 years ago by Jens Axboe and probably developed only for drivers from that time. Maybe you can try to contact him if he remember details.

Anyway I have looked into git history and code. USE_IMMED was changed from 1 to 0 in commit 5501a92b0c9f2f3e5edd5f837b0c353b7b1be0c3. And seems that cdrwtool code is prepared from using immed bit when USE_IMMED is set to 1. Then also print_completion_info and sig_progress should be used.

scdbackup commented 3 years ago

Hi,

Maybe you should use WAIT_BLANK macro (or derivate from it real time interval, it is 60 minutes) instead of 20 minutes. As currently format_disc() for GPCMD_FORMAT_UNIT command is using WAIT_BLANK timeout.

Ok. I'll coordinate wait_for_unit_ready() with that timeout.

Or cannot you set USE_IMMED bit?

When was this last tested ? It looks multiply broken.

I now found the TEST UNIT READY code of cdrwtool: sig_progress(), the function which believes that wait_cmd() would fill a submitted struct request_sense with the reply from the SCSI command. It does not, but rather uses its own local struct request_sense.

The sig_progress() mechanism uses asynchronous processing rather than synchronous waiting. Currently wait_cmd() returns error if CDROM_SEND_PACKET returns sense data. If the sense return would work, it should at least cause ugly messages if not early end of waiting. (I had to silence wait_cmd() in case of GPCMD_TEST_UNIT_READY.)

The waiting loop in print_completion_info() ends when it thinks that the progress indicator is high enough, not when TEST UNIT READY indicates readiness. The progress indicator is a nice-to-have, not something we can rely on. The code bails out early if it thinks that progress is not available. It then says "Don't access drive until operation has completed\n" A demand which the code in quick_setup() neither sees nor obeys.

Honestly, i think this way of waiting is overly complicated and invites either premature end with subsequently failing SCSI commands or endless looping. It depends on which visible mistake gets precedence.

Would you mind if i replace the signal()-alarm()-while() construct in print_completion_info() by a call to my synchronous wait_for_unit_ready() and let it emit "xy% complete message" like in sig_progress(). sig_progress() itself and the global variable "progress" would then be obsolete.

New drivers do not accept requirements from MMC-1 and old drivers obviously do not support MMC-7 (when was only MMC-1 present).

2 of 3 new-ish drives support the old format. After all it is well distinguishable from the new one.

(A bit more flaky is the use of default IP bit setting, which controls the presence of the 4 extra bytes between format list header and format descriptor. With new Code 1, IP is prescribed to be 0. But in Code 7 of MMC-1, an IP value of 1 seems to be the default.)

Do we know other applications handles this issue? If yes, we should do the same thing.

libburn uses the new Code 1. I was not aware of Code 7 until yesterday. dvd+rw-format uses Code 1. (I learned much about DVD burning from dvd+rw-tools.)

If not, I'm for issuing new MMC-7 way with fallback to old MMC-1.

Ok. Will work in that direction.

Jens Axboe [...] Maybe you can try to contact him if he remember details.

He may have been active in that field before me. But since 2006 i am quite surely more busy in it than he is. :))

Best i can hope for is that he commits flawless kernel patches, which i still have to learn to create. https://lore.kernel.org/linux-scsi/20201006094026.1730-1-scdbackup@gmx.net/T/#u https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbackup@gmx.net/T/#u are obviously not good enough yet.

USE_IMMED was changed from 1 to 0 in commit 5501a92

At that time, the sense code transmission between sig_progress() and wait_cmd() was already broken.

Have a nice day :)

Thomas

callegar commented 3 years ago

@scdbackup Tried the proposed tests on the Maxell BD recorder with a CDRW:

  1. xsorriso seems to work and finally eject just fine, even thought there are a few weird things:
    • It reports xorriso : NOTE : Disc status unsuitable for writing at the very beginning
    • While blanking it keeps saying 1.0% done all the time without progressing as in 
      xsorriso : UPDATE : Blanking  ( 1.0% done in 1 seconds )
xorriso : UPDATE : Blanking  ( 1.0% done in 2 seconds )
xorriso : UPDATE : Blanking  ( 1.0% done in 3 seconds )
    • In fact, notwithstanding the lack of increasing percentages the blanking gets to an end...
    • After the blanking it starts writing, but the writing seems to be split in two parts at 4MB: 
      Beginning to write data track.
xorriso : UPDATE :    0 MB written (fifo 99%) [buf   0%]
xorriso : UPDATE :    0 MB written (fifo 99%) [buf   0%]   0.0x.
xorriso : UPDATE :    0 MB written (fifo 99%) [buf   0%]   0.0x.
xorriso : UPDATE :    1 MB written (fifo 99%) [buf  84%]   7.5x.
<... more lines like above...>
xorriso : UPDATE :    1 MB written (fifo 98%) [buf  95%]   3.0x.
xorriso : UPDATE :    3 MB written (fifo 99%) [buf  94%]  10.0x.
xorriso : UPDATE :    4 MB written (fifo 99%) [buf  97%]  10.0x.
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 52,5009 s, 200 kB/s
xorriso : UPDATE :    5 MB written (fifo 99%) [buf  94%]  10.0x.
xorriso : UPDATE :    7 MB written (fifo 64%) [buf  96%]  10.0x.
xorriso : UPDATE :    8 MB written (fifo 27%) [buf  96%]  10.0x.
<... more lines like above...>
xorriso : UPDATE :   10 of   10 MB written (fifo  0%) [buf  96%]   0.0x.
    • Closing the medium takes a good 63 seconds
  2. cdrwtoool starts as you say, with the I'm going to do a quick setup of /dev/sr0 etc., but then errors out:
    • It seems to format the medium fine 
      Initiating quick disc blank
Disc capacity is 295264 blocks (590528KB/576MB)
Formatting track
start=0, blocks=16, type=RESERVED 
start=16, blocks=4, type=VRS 
start=20, blocks=76, type=USPACE 
start=96, blocks=32, type=MVDS 
start=128, blocks=32, type=LVID 
start=160, blocks=5, type=STABLE 
start=165, blocks=91, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=31, type=USPACE 
start=288, blocks=1024, type=SSPACE 
start=1312, blocks=293664, type=PSPACE 
start=294976, blocks=31, type=USPACE 
start=295007, blocks=1, type=ANCHOR 
start=295008, blocks=96, type=USPACE 
start=295104, blocks=32, type=RVDS 
start=295136, blocks=96, type=USPACE 
start=295232, blocks=5, type=STABLE 
start=295237, blocks=26, type=USPACE 
start=295263, blocks=1, type=ANCHOR
    • but then it fails at writing the UDF structures to disk 
      Writing UDF structures to disc
wait_cmd: Input/output error
Command failed: 2a 00 00 00 00 00 00 00 20 00 00 00 - sense 02.04.04
wait_cmd: Input/output error
Command failed: 2a 00 00 00 00 00 00 00 20 00 00 00 - sense 02.04.04  
wait_cmd: Input/output error
Command failed: 35 02 00 00 00 00 00 00 00 00 00 00 - sense 02.04.04
Quick setup complete!
scdbackup commented 3 years ago

Hi,

[xorriso]

  • It reports xorriso : NOTE : Disc status unsuitable for writing at the very beginning

Then the CD was already closed.

 +  While blanking it keeps saying 1.0% done all the time without
    progressing as in

Then the drive does not tell progress information when TEST UNIT READY asks whether we are already there. But it tells when it is finally ready.

  1. cdrwtoool starts as you say ... Formatting track

At least it came beyond the error occasion with 55 MODE SELECT.

Strange. Why didn't it without the xorriso run, when CD-RW was already closed ?

start=0, blocks=16, type=RESERVED

Function format_disc() has reported success.

Do i guess right that it did not last longer than 10 seconds ?

Command failed: 2a 00 00 00 00 00 00 00 20 00 00 00 - sense 02.04.04

But the FORMAT UNIT command is not done yet. Like my Pioneer BD burner it seems to act as if the Immed bit was set. Probably the drive will be busy for a bit less than 10 minutes.

I am working on the problem of premature writing and will soon ask pali to give me hands-on instructions how to submit a lengthy patch. git diff currently yields 405 lines. After some decisions i expect about 300 to remain.

Quick setup complete!

This is a very optimistic end message. One should investigate how writing can fail without being noticed by the upper layers.

So the trick seems to work, to burn a sequential session to a CD-RW and to close it if cdrwtools refuses on command 55 MODE SELECT.

This can be done by cdrecord, wodim, or "xorriso -as cdrecord" by the same options. Of course the GUI programs K3B, Xfburn, Brasero can achieve the same. Just make sure that the medium is not kept appendable for more sessions.

Have a nice day :)

Thomas

callegar commented 3 years ago

@scdbackup I now realize that I must have used the same CDWR that I employed in the previous tests and that probably in a recent past I had formatted with cdrwtool using the other DVD writer I have, namely an LG portable GP08NU20 (which happens to work with cdrwtool), to make sure the disk was not faulty. But I believe that was OK.

Do i guess right that it did not last longer than 10 seconds ?

I have not taken an exact measurement, but the cdrwtool run ending up with the error was quite fast.

scdbackup commented 3 years ago

Hi,

it seems i have implemented the features which make my three test drives succeed. USE_IMMED == 1 works for blanking and formatting and causes entertaining percentage messages every 5 seconds. (Still quite a flood when formatting lasts 500 seconds.)

...
Initiating quick disc blank
19% complete
48% complete
77% complete
100% complete
100% complete and done: BLANK
Disc capacity is 295264 blocks (590528KB/576MB)
Formatting track
00% complete
01% complete
02% complete
...
94% complete
94% complete
100% complete and done: FORMAT UNIT 1
...

The Pioneer drive counts backward with blanking and shows 0% with formatting until the very end. (Shrug.)

The change consists of:

@pali: Is it necessary to prepare three separate commits/patches ? How will i submit it/them ?

Have a nice day :)

Thomas

pali commented 3 years ago

@scdbackup: Try to split logical changes into separate commits and feel free to open a pull request. Commenting code is easier in pull request where can be put "inline" comments.

pali commented 3 years ago

(reply to the first comment)

Command failed: bb 00 ff ff 08 40 00 00 00 00 00 00 - sense 00.00.00 set speed

BB is SET SPEED. Sense 0,0,0 looks like not the drive but the kernel raised protest. (One would have to print SG_IO diagnostics like struct sg_io_hdr_t.host_status and sg_io_hdr_t.driver_status. But cdrwtool.c still uses elderly CDROM_SEND_PACKET instead of SG_IO.)

I'm looking at the kenel code and it seems that CDROM_SEND_PACKET is just transformed to SG_IO. It is possible to use CDROM_SEND_PACKET for any usb, sata or scsi disk (/dev/sd*). E.g. INQUIRY command via CDROM_SEND_PACKET is successfully executed also for /dev/sda disk.

And second observation of kernel code, it looks like that sense is filled only when ioctl returns EIO error set in errno variable. So maybe we should update debug code to print also errno as in case it is not EIO then in sense is probably not touched at all. Userspace program initialize memory in sense array to zeros and therefore these 00.00.00 are in output.

scdbackup commented 3 years ago

Hi,

cdrwtool.c still uses elderly CDROM_SEND_PACKET instead of SG_IO.

I'm looking at the kenel code and it seems that CDROM_SEND_PACKET is just transformed to SG_IO. It is possible to use CDROM_SEND_PACKET for any usb, sata or scsi disk (/dev/sd*).

Yes. But it obscures the reply information of https://tldp.org/HOWTO/SCSI-Generic-HOWTO/x291.html https://tldp.org/HOWTO/SCSI-Generic-HOWTO/x322.html

If you get an error and sense 0,0,0 you can only guess that something outside the drive firmware went wrong. Especially sg_io_hdr.host_status can be of help when the hardware goes bad.

Have a nice day :)

Thomas

scdbackup commented 3 years ago

Hi,

i have now prepared two changesets:

The first one touches wait_cmd(). So please refrain from changing the handling of CDROM_SEND_PACKET before you have decided about my change.

The second gets as patch some complaints from Linux kernel scripts/checkpatch.pl. They all stem from lines of the existing code which i copied or modified. Following the advise of the script would create code style which would stand out in the existing context:

ERROR: that open brace { should be on the previous line
#62: FILE: cdrwtool/cdrwtool.c:465:
+       if ((ret = wait_cmd(fd, &cgc, buffer, CGC_DATA_WRITE, WAIT_BLANK)) < 0)
+       {

ERROR: do not use assignment in if condition
#62: FILE: cdrwtool/cdrwtool.c:465:
+       if ((ret = wait_cmd(fd, &cgc, buffer, CGC_DATA_WRITE, WAIT_BLANK)) < 0)

WARNING: function definition argument 'int' should also have an identifier name
#112: FILE: cdrwtool/cdrwtool.h:213:
+int format_disc(int, struct cdrw_disc *, write_params_t *);

... 2 more warnings about the other tow parameters ...

ERROR: do not use assignment in if condition
#135: FILE: cdrwtool/main.c:173:
+               if ((ret = format_disc(fd, disc, w)))

Shall i fix those complaints at the cost of making coding style more uneven than it is now ?

Still a diffuse problem:

I cannot find usable instructions how to create the preconditions for a github pull request. The many tutorials in the web show GUI elements which i do not see in my browser.

Is there a turorial known which works for old people with an old Debian Iceweasel browser ? (Or is there a way to do this with command line activities ?)

Have a nice day :)

Thomas

pali commented 3 years ago

The second gets as patch some complaints from Linux kernel scripts/checkpatch.pl.

udftools uses different coding style than Linux kernel, so errors from scripts/checkpatch.pl are not relevant for udftools in most cases. So ignore them.

When dealing with coding style in udftools, please use common sense and see what is actual coding style of currently editing file. I think it could be obvious to determinate it.

I cannot find usable instructions how to create the preconditions for a github pull request. The many tutorials in the web show GUI elements which i do not see in my browser.

Is there a turorial known which works for old people with an old Debian Iceweasel browser ?

Well, everthing on web now needs up-to-date web browser and only Firefox (and their clones) and Chrome (and their clones) are supported.

First you need to create personal fork of my udftools repository (it will appear in your account). This can be done by selecting your account (where your personal fork would be created) at page:

https://github.com/pali/udftools/fork

Then via git push push changes into your personal fork (either via ssh or https protocols, depending if you configured on github ssh key or just plaintext auth in https). And after that create a pull request. This can be done by opening following URL (assuming your personal repository is named udftools and branch with your changes is master):

https://github.com/scdbackup/udftools/pull/new/master

(Or is there a way to do this with command line activities ?)

If you like a command line you can try to use official github command line tool gh:

https://cli.github.com/manual/gh_repo_fork https://cli.github.com/manual/gh_pr_create https://cli.github.com/manual/

(NOTE: I have never used it, so cannot help with it)

pali commented 3 years ago

For command line there is also hub tool: https://hub.github.com/hub-fork.1.html https://hub.github.com/hub-pull-request.1.html https://hub.github.com/

pali commented 3 years ago

I have tried above hub tool (it is available in Debian hub package) and it is working fine. I can create both repositories and pull requests. It is just required to create an access token at this page https://github.com/settings/tokens/new, enable at least public_repo scope and use this token as password when hub is asking for password. It is because github dropped password-login support in api tools and only tokens are working.

scdbackup commented 3 years ago

Hi,

sorry, i'm having health problems and had to interrupt Github learning yesterday after installing Debian package "hub".

I now got a token from https://github.com/settings/tokens/new with only "public_repo". But i'm feeling currently not able to go ahead. Hopefully i will be in better shape in a few days.

So for now i attach my patches to keep them from getting lost. This does not mean that i don't strive for a pull request. Just not today.

While polishing my patches yesterday i could not stop myself from diagnosing the last complaint of cdrwtool with my drives: can't unlock door

It happens because in Linux drivers/cdrom/cdrom.c the function cdrom_ioctl_lock_door() imposes a ban on unlocking if the drive is in use by others and the user has no CAP_SYS_ADMIN. The second user is the pktcdvd device which is connected to the sr device. I can get rid of the complaint if i am superuser or if i delete the pktcdvd device. But that's of course no solution for production.

My proposal would be to give up the calls of ioctl(fd, CDROM_LOCKDOOR, ) and to replace them by wait_cmd() with PREVENT ALLOW MEDIUM REMOVAL commands. Lock the door: 1e 00 00 00 01 00 Unlock: 1e 00 00 00 00 00 (With my dizzy brain i rely on libburn's SCSI log. One should verify the bytes by reading MMC specs.)

Have a nice day :)

Thomas

scdbackup commented 3 years ago

Hi,

well, the attaching seems not to have worked. Duh.

Falling back to the technology of the 20th century:

http://scdbackup.sourceforge.net/0001-cdrwtool-Revive-USE_IMMED.patch http://scdbackup.sourceforge.net/0001-cdrwtool-Switch-to-FORMAT-UNIT-Format-Code-1.patch

They were tested with 3 halfways modern drives. The fallback to the old Format Code 7 is supposed to be necessary with drives which are at least 10 years old. MMC-3 is from 2001, MMC-4 introduced the new Format Code in 2004.

So if somebody here has a really old drive, please test. (I planned to make mock-up tests. But well ...)

Have a nice day :)

Thomas

pali commented 3 years ago

Take a free time to be a healthy! No need to hurry.

pali commented 3 years ago

When you are healthy and back again and still have issues with command line hub command, I can try to help with it.

scdbackup commented 3 years ago

Hi,

after the snot level sunk below my eyebrows i gave the pull request a try:

export GITHUB_TOKEN=...lengthy.hex.string.from.github...
hub clone pali/udftools udftools-hub
cd udftools-hub
hub fork

git apply ../cdrwtool_patches/0001-cdrwtool-Revive-USE_IMMED.patch
git add cdrwtool/cdrwtool.h cdrwtool/cdrwtool.c
git commit -F ...commit.message.file...

git apply ../cdrwtool_patches/0001-cdrwtool-Switch-to-FORMAT-UNIT-Format-Code-1.patch
git add cdrwtool/main.c cdrwtool/cdrwtool.h cdrwtool/cdrwtool.c
git commit -F ...commit.message.file...

./autogen.sh
./configure && make

Then i tested again with all three available drives:

# Bring the CD-RW into written and closed state. Then eject it:
dd if=/dev/zero bs=1M count=10 | \
  xorriso -as cdrecord -v -eject dev=/dev/sr2 blank=as_needed -
# Load the tray again with proper waiting (unlike the kernel does):
xorriso -outdev /dev/sr2
# Format
./cdrwtool/cdrwtool -d /dev/sr2 -q

(With the Pioneer BD-RW BDR-S09 counting percentage backwards during the blanking stage and obstinately reporting 0 percent progress while formatting until it is done. It reports similar trash with libburn too.)

But now i am stuck in the swamp:

$ hub pull-request -m 'For issue 33'
Aborted: 2 commits are not yet pushed to scdbackup/master
(use `-f` to force submit a pull request anyway)

$ hub pull-request -p -m 'For issue 33'
Aborted: 2 commits are not yet pushed to scdbackup/master
(use `-f` to force submit a pull request anyway)

$ git push
fatal: remote error: 
You can't push to git://github.com/pali/udftools.git
Use https://github.com/pali/udftools.git

What did i do wrong ?

Have a nice day :)

Thomas

pali commented 3 years ago

hub fork

Your personal repository now exists! https://github.com/scdbackup/udftools

$ hub pull-request -m 'For issue 33' Aborted: 2 commits are not yet pushed to scdbackup/master

You first need to push your local changes into your personal repository via git push.

$ git push fatal: remote error: You can't push to git://...

git:// protocol does not support push operation. You need to use https:// protocol with github password or ssh protocol with private key (public must be configured in your github account).

See .git/config file where you can configure protocol.

Use https://github.com/pali/...

You cannot push changes into my profile :-) You need to push it into your personal fork. Pull request is then creating from your personal fork.

scdbackup commented 3 years ago

Hi,

You cannot push changes into my profile :-)

This is plausible. But how do i change the push target ?

You need to use https:// protocol with github password See .git/config file where you can configure protocol.

Well i could change "git://github.com/pali/udftools.git" to "https://" but that will probably not solve the push target problem. Currently i see in .git/config

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git://github.com/pali/udftools.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[remote "scdbackup"]
        url = git@github.com:scdbackup/udftools.git
        fetch = +refs/heads/*:refs/remotes/scdbackup/*

Do you have proposals how to fix it ? (I can well start from scratch if that is easier. In this case tell me which hub commands i shall run.)

Have a nice day :)

Thomas

pali commented 3 years ago

You already have defined remote for your personal repository. Each git branch which you have can be configured to which repository is pushed by git push command if you do not specify any argument for git push. I guess that in your case you are currently on master branch and it has configured that default repository is origin. Normally when using git you first create a new (local) branch, switch into it and then do all changes in that branch.

How to push your local changes to your scdbackup remote repository? Just explicitly specify to git push command that you want to push changes into your scdbackup remote. E.g.:

$ git push scdbackup

(this will push current local branch to remote branch in repository scdbackup)

Or full syntax is:

$git push <name_of_remote/full_url> <local_branch/local_commit>:<remote_branch>

(full syntax allows to push changes from any branch, not only currently selected and allow to push into any remote branch; when using full syntax, in some cases needs to be prefixed by refs/heads/)