Random "Cannot write index/EOM" errors when unmounting sync_type=unmount tape

[Azenet]

Describe the bug Sometimes, when unmounting a cartridge (mounted with sync_type=unmount) that has been completely filled so it has dropped to read-only, the index will fail to write because the drive seems to reach the end of the partition/medium. I have compression enabled, although I'm writing only files that cannot be much compressed by the drive.

To Reproduce Steps to reproduce the behavior:

1. Create a tape, mount it with -o sync_type=unmount and write files to it until the tape is full and drops to read-only
2. Unmount the tape

Expected behavior The index is written successfully and the tape is unmounted.

Unmount logs

``` f5479 LTFS17235I Writing index of LT0011 to b (Reason: Unmount, 503 files) xxx. f5479 LTFS30222W Received low space warning (early warning) in write filemarks. f5479 LTFS30222W Received low space warning (early warning) in write. f5479 LTFS30222W Received low space warning (early warning) in write. f5479 LTFS30222W Received low space warning (early warning) in write. [above repeated 50 times] f5479 LTFS30205I WRITEFM (0x10) returns -21300. f5479 LTFS30263I WRITEFM returns End-of-Partition/Medium Detected (-21300) /dev/sg1. f5479 LTFS12047E Cannot write file marks: backend call failed (-21300). Dropping to read-only mode. f5479 LTFS11084E Cannot write index: failed to write a trailing file mark (-21300). The medium is in an inconsistent state; unmount and run ltfsck. f5479 LTFS11083E Cannot write index: failed to generate and write XML data (-21300). The medium might be in an inconsistent state; unmount and run ltfsck. f5479 LTFS17235I Writing index of LT0011 to a (Reason: Unmount, 503 files) xxx. f5479 LTFS11082E Cannot write index: failed to write file mark (-1126). f5479 LTFS11033E Cannot unmount: failed to write an index. f5479 LTFS30205I MODESENSE (0x5a) returns -20501. f5479 LTFS30263I MODESENSE returns Invalid Field in CDB (-20501) /dev/sg1. ```

Successful mount logs

``` 1e1055 LTFS14000I LTFS starting, LTFS version 2.4.4.0 (10470), log level 2. 1e1055 LTFS14058I LTFS Format Specification version 2.4.0. 1e1055 LTFS14104I Launched by "ltfs -o devname=/dev/sg1 -o sync_type=unmount -f /mnt/ltfs/". 1e1055 LTFS14105I This binary is built for Linux (x86_64). 1e1055 LTFS14106I GCC version is 11.1.0. 1e1055 LTFS17087I Kernel version: Linux version 5.10.84-1-lts (linux-lts@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP Wed, 08 Dec 2021 10:17:01 +0000 i386. 1e1055 LTFS17089I Distribution: NAME="Arch Linux". 1e1055 LTFS17089I Distribution: LSB_VERSION=1.4. 1e1055 LTFS17089I Distribution: . 1e1055 LTFS14064I Sync type is "unmount". 1e1055 LTFS17085I Plugin: Loading "sg" tape backend. 1e1055 LTFS17085I Plugin: Loading "unified" iosched backend. 1e1055 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection. 1e1055 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg1). 1e1055 LTFS30250I Opened the SCSI tape device 6.0.0.0 (/dev/sg1). 1e1055 LTFS30207I Vendor ID is HP . 1e1055 LTFS30208I Product ID is Ultrium 5-SCSI . 1e1055 LTFS30214I Firmware revision is I57H. 1e1055 LTFS30215I Drive serial is xxx. 1e1055 LTFS30285I The reserved buffer size of /dev/sg1 is 1048576. 1e1055 LTFS30294I Setting up timeout values from RSOC. 1e1055 LTFS17160I Maximum device block size is 1048576. 1e1055 LTFS11330I Loading cartridge. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS11332I Load successful. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS17157I Changing the drive setting to write-anywhere mode. 1e1055 LTFS11005I Mounting the volume. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS30205I MODESENSE (0x5a) returns -20501. 1e1055 LTFS30263I MODESENSE returns Invalid Field in CDB (-20501) /dev/sg1. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS39805W The timeout table is not configured: SCSI OP code 0x4d. Use the default timeout. 1e1055 LTFS17227I Tape attribute: Vendor = IBM . 1e1055 LTFS17227I Tape attribute: Application Name = LTFS . 1e1055 LTFS17227I Tape attribute: Application Version = 2.4.4.0 . 1e1055 LTFS17227I Tape attribute: Medium Label = . 1e1055 LTFS17228I Tape attribute: Text Localization ID = 0x81. 1e1055 LTFS17227I Tape attribute: Barcode = LT0011 . 1e1055 LTFS17227I Tape attribute: Application Format Version = 2.4.0 . 1e1055 LTFS17228I Tape attribute: Volume Lock Status = 0x00. 1e1055 LTFS17227I Tape attribute: Media Pool name = . 1e1055 LTFS11031I Volume mounted successfully. LT0011 : Gen = 3 / (a, 5) -> (b, 2975316) / xxx. 1e1055 LTFS14111I Initial setup completed successfully. 1e1055 LTFS14112I Invoke 'mount' command to check the result of final setup. 1e1055 LTFS14113I Specified mount point is listed if succeeded. 1e1696 LTFS14029I Ready to receive file system requests. ```

Additional context The index can sometimes be recovered by running ltfsck -l, which rewrites the index and may succeed straight away. I have not reproduced the issue with default sync_type for now, however I'm not sure whether the issue may only happen with sync_type=unmount. My drive is a HP Ultrium 5-SCSI FH with firmware I57H, the tape make this last happened with is Quantum.

Version ltfsck version 2.4.4.0 (10470). LTFS Format Specification version 2.4.0.

Thank you for any help!

[piste-jp]

Interesting.

It looks LTFS receives early warning at writing the first FM of an index. And this HP drive returns end-of-medium just after 50 writes when early warning is reported. LTFS writes a record with 512KiB. So it means HP drive reports early warning just before 30 MiB at most.

1. Could you provide a the log before f5479 LTFS17235I Writing index of LT0011 to b (Reason: Unmount, 503 files) xxx. ? I need to check the second line of your unmount log is the first report of early warning.
2. It looks your index is slightly long. Do you know why the index has such capacity?
   • Do you use long file name?
   • Do you store something into extended attribute?

[Azenet]

These are the log lines printed when the write fails due to the tape being full and the tape goes to read-only:

f5649 LTFS30222W Received low space warning (early warning) in write.
f5649 LTFS11072E Cannot write blocks: failed to write to the medium (-1051).
f5649 LTFS11077E Cannot write: failed to write blocks to the medium (-1051).
f5649 LTFS13014W Data partition writer: failed to write data to the tape (-1051).

Then the unmount log starts when I umount the tape.

Filenames are 45 characters long on average and files have no extended attributes. There are 503 files on this specific tape as it was filled in one go.

[piste-jp]

Thank you for your addition info.

It looks reporting timing of early warning on the HP's LTO5 is too late. We need to have 500MiB at least...

On IBM tape drives, they have programmable early warning feature and the LTFS set it (if it is available) to 20GiB for spare capacity to write index with meta-data modification later. So this kind of error doesn't happen at all. We can introduce equivalent functionality in the future but it might cause performance drop in non-IBM drives. (The idea id checking remaining capacity at each 10GiB writes or something...)

[Azenet]

Thanks for the information. I have since opening this issue written 7 Fujifilm and HP tapes and have not had this issue (similar filenames and file count), so I'm not sure whether this issue only reproduces with Quantum tapes. I want to add the tape I had this issue with was on its first write after being formatted.

[piste-jp]

Generally, tape has a kind of spare capacity to write down data to meet the LTO spec. And usually, LTO drive returns early warning when some amount of data is written.

So drive might return early warning at the very close to the tape end, if write quality is not good. I guess your quantum tapes would not have a good performance against your tape. R/W quality stuff is so much complicated, I recommend to use HP's media because they might adjust their tape head to their tape.

I know LTO has a kind of compatibility but it doesn't mean every drives can use every tapes with the best quality.

[piste-jp]

Close this.