Closed mivk closed 5 years ago
I never see such kind of error before. I think sg driver or lower transfer layer driver returns an error because errno=22
.
#define EINVAL 22 /* Invalid argument */
At the format time, there is no need to write 512KB record on the tape. (LTFS just writes 80-bytes VOL-1, shot records (< 1K) and FMs). But LTFS will try to write 512KB record at writing teh first file on LTFS (if the file is larger than 512KB).
-b 131072
)? Below is the HBA/tape relevant part of the lshw
output. I could try the -b 131072
formatting switch tomorrow.
In the meantime, I tried to recompile with --enable-lintape
, and using the lin_tape driver instead of sg. That works as expected, and the tape is being written to now. But of course, I would much prefer not having to use that horribly cumbersome IBM lin_tape module.
So I hope it is possible to make it work with the standard sg module.
Here is the HBA info, with the tape drive and changer attached to it:
*-sas
description: Serial Attached SCSI controller
product: ExpressSAS 6Gb/s SAS/SATA HBA
vendor: ATTO Technology, Inc.
physical id: 0
bus info: pci@0000:81:00.0
logical name: scsi9
version: 00
width: 64 bits
clock: 33MHz
capabilities: sas pm msi pciexpress msix bus_master cap_list rom
configuration: driver=pm80xx latency=0
resources: irq:76 memory:fb230000-fb23ffff memory:fb220000-fb22ffff memory:fb210000-fb21ffff memory:fb200000-fb20ffff memory:fb100000-fb1fffff
*-tape UNCLAIMED
description: SCSI Tape
product: ULTRIUM-HH7
vendor: IBM
physical id: 0.1.0
bus info: scsi@9:0.1.0
version: J4D1
serial: 10WT064673
capabilities: removable
configuration: ansiversion=6
*-changer UNCLAIMED
description: SCSI Medium Changer
product: FlexStor II
vendor: BDT
physical id: 0.1.1
bus info: scsi@9:0.1.1
version: 5.50
serial: 00MX64002098_LL0
capabilities: removable
configuration: ansiversion=5
Hmm, that's interesting...
In the meantime, I tried to recompile with --enable-lintape, and using the lin_tape driver instead of sg. That works as expected, and the tape is being written to now. But of course, I would much prefer not having to use that horribly cumbersome IBM lin_tape module.
Actually I didn't try any ATTO HBAs on Linux yet.
I suggested to try mkltfs
with -b 131072
. But I found more effective way. Could you check how many length can be transferred by sg + ATTO combination in the following procedure?
dd if=/dev/zero of=/mnt/ltfs/dummy_32 bs=32K count=1
dd if=/dev/zero of=/mnt/ltfs/dummy_64 bs=64K count=1
dd if=/dev/zero of=/mnt/ltfs/dummy_128 bs=128K count=1
dd if=/dev/zero of=/mnt/ltfs/dummy_256 bs=256K count=1
dd if=/dev/zero of=/mnt/ltfs/dummy_512 bs=512K count=1
Apparently, 512K is the maximum. Trying to write more than that fails and the tape becomes read-only.
This worked:
# for s in 32 64 128 256 512; do dd if=/dev/zero of=/mnt/ltfs/dummy_$s bs=${s}K count=1 && echo OK $s K || break; done
1+0 records in
1+0 records out
32768 bytes (33 kB) copied, 6.05268 s, 5.4 kB/s
OK 32 K
1+0 records in
1+0 records out
65536 bytes (66 kB) copied, 0.0015538 s, 42.2 MB/s
OK 64 K
1+0 records in
1+0 records out
131072 bytes (131 kB) copied, 0.00184851 s, 70.9 MB/s
OK 128 K
1+0 records in
1+0 records out
262144 bytes (262 kB) copied, 0.00279722 s, 93.7 MB/s
OK 256 K
1+0 records in
1+0 records out
524288 bytes (524 kB) copied, 0.00415381 s, 126 MB/s
OK 512 K
But trying to increase the count to 2 failed at 2 x 512K:
# for s in 32 64 128 256 512; do dd if=/dev/zero of=/mnt/ltfs/dummy_$s-x2 bs=${s}K count=2 && echo OK $s K || break; done
2+0 records in
2+0 records out
65536 bytes (66 kB) copied, 0.0635124 s, 1.0 MB/s
OK 32 K
2+0 records in
2+0 records out
131072 bytes (131 kB) copied, 0.00185673 s, 70.6 MB/s
OK 64 K
2+0 records in
2+0 records out
262144 bytes (262 kB) copied, 0.00237088 s, 111 MB/s
OK 128 K
2+0 records in
2+0 records out
524288 bytes (524 kB) copied, 0.00401603 s, 131 MB/s
OK 256 K
dd: error writing ‘/mnt/ltfs/dummy_512-x2’: Read-only file system
2+0 records in
1+0 records out
786432 bytes (786 kB) copied, 0.00461646 s, 170 MB/s
After that, nothing could be written to the tape.
Thank you for your trial, I think first write of 512KB is failed. And 1st 512KB file is failed to write, but LTFS does deferred write so 1st write of 512KB file looks successfully done. This is a very natural behavior of modern filesystem.
I think you can use 256KB block in the sg backend. The block size is stored into the LTFS label and LTFS index. So another LTFS implementation shall support that.
I strongly recommend to format tapes with mkltfs -b 262144
for a while to avoid this problem. And I believe performance impact is very small to change blocksize from 512KB to 256KB.
On the other hand we need to investigate this HBA. Do you know the type of this HBA? (output of dmesg
is useful I think.)
Yes, with mkltfs -b 262144
, it works with the sg driver!
It would be nice to have it also work with the default block size, but in the meantime this is a good workaround.
The ATTO HBA is the ExpressSAS H680 model.
There doesn't seem to be anything interesting in dmesg
output, but here is the relevant output of lspci -vv
: lspci-atto-h680.txt
Thank you very much.
I have one H680 for Mac. I will try that in my spare time (Sorry I'm little bit busy now).
And could you provide a output of following commands after LTFS mounts a tape in addition?
cat /proc/scsi/sg/allow_dio
cat /proc/scsi/sg/debug
Here is the output, while the tape is being written to:
$ cat /proc/scsi/sg/allow_dio
1
$ cat /proc/scsi/sg/debug
max_active_device=9 def_reserved_size=1048576
>>> device=sg4 0:0:0:0 em=0 sg_tablesize=128 excl=1 open_cnt=1
FD(1): timeout=60000ms bufflen=524288 (res)sgat=16 low_dma=0
cmd_q=1 f_packid=0 k_orphan=0 closed=0
dio>> act: id=0 blen=0 t_o/elap=1560000/196ms sgat=0 op=0x0a
Thank you very much ! I realized DIO is enabled.
I think default setting is disable. And I hit multiple problems when DIO is enabled on Emulex FC HBA too (See https://github.com/LinearTapeFileSystem/ltfs#note-for-the-lpfc-driver-emulex-fibre-hbas). So it is not a surprise if there is a problem in ATTO. At this time, I know QLogic FC HBA is the only HBA the DIO works correctly on 512KB transfer.
I don't know you can accept to change the DIO setting disabled. But I guess you can use 512KB block length if you disable DIO.
Yes, I had added it after reading Parameter settings of the sg driver. Will try to remove it when I get a chance and report back (some time next week).
I'm seeing a similar error message ("Failed to execute SG_IO ioctl, opcode = 0a (22)."), but I'm not sure if I'm seeing the same root problem or not.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14000I LTFS starting, LTFS version 2.4.1.1 (10227), log level 3.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14058I LTFS Format Specification version 2.4.0.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14104I Launched by "ltfs -o devname=/dev/st0 /opt/VS/vsMounts/000025L7 -o trace".
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14105I This binary is built for Linux (x86_64).
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14106I GCC version is 6.3.0 20170516.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS17087I Kernel version: Linux version 4.20.11 (root@debian) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Thu Feb 21 09:13:14 EST 2019 i386.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 9 (stretch)".
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14063I Sync type is "time", Sync time is 300 sec.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS17085I Plugin: Loading "sg" tape backend.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS17085I Plugin: Loading "unified" iosched backend.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS30209I Opening a device through sg-ibmtape driver (/dev/st0).
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS30207I Vendor ID is IBM .
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS30208I Product ID is ULTRIUM-HH7 .
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS30214I Firmware revision is JAY1.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS30215I Drive serial is 10WT074468.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS17160I Maximum device block size is 1048576.
Sep 3 11:48:44 alice2 ltfs[25370]: 631a LTFS11330I Loading cartridge.
Sep 3 11:48:47 alice2 ltfs[25370]: 631a LTFS30252I Logical block protection is disabled.
Sep 3 11:48:47 alice2 ltfs[25370]: 631a LTFS11332I Load successful.
Sep 3 11:48:47 alice2 ltfs[25370]: 631a LTFS17157I Changing the drive setting to write-anywhere mode.
Sep 3 11:48:47 alice2 ltfs[25370]: 631a LTFS11005I Mounting the volume.
Sep 3 11:48:47 alice2 ltfs[25370]: 631a LTFS30252I Logical block protection is disabled.
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Vendor = IBM .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Name = LTFS .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Version = 2.4.1.1 .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Medium Label = .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17228I Tape attribute: Text Localization ID = 0x81.
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Barcode = 000025L7 .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Format Version = 2.4.0 .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17228I Tape attribute: Volume Lock Status = 0x00.
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Media Pool name = .
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS14111I Initial setup completed successfully.
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS14112I Invoke 'mount' command to check the result of final setup.
Sep 3 11:49:01 alice2 ltfs[25370]: 631a LTFS14113I Specified mount point is listed if succeeded.
Sep 3 11:49:01 alice2 ltfs[25412]: 6345 LTFS14029I Ready to receive file system requests.
Sep 3 11:49:03 alice2 ltfs[25412]: 6346 LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x5579c4466410).
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS30200I Failed to execute SG_IO ioctl, opcode = 0a (22).
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS30263I WRITE returns ioctl error (-21700) /dev/st0.
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS12045E Cannot write block: backend call failed (-21700). Dropping to read-only mode.
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS11072E Cannot write blocks: failed to write to the medium (-21700).
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS11077E Cannot write: failed to write blocks to the medium (-21700).
Sep 3 11:49:41 alice2 ltfs[25412]: 6347 LTFS13014W Data partition writer: failed to write data to the tape (-21700).
Sep 3 11:49:41 alice2 ltfs[25412]: 6345 LTFS14022E Cannot flush file data to the medium. Data might be lost (ltfs_fuse_flush).
Sep 3 11:49:41 alice2 ltfs[25412]: 6346 LTFS10020E Error on utimens: /Preview+Archive.24/2019-08-31/h264/0720_06_59.mkv (0, 0).
Sep 3 11:49:45 alice2 ltfs[25412]: 6344 LTFS11033E Cannot unmount: failed to write an index.
Sep 3 11:49:45 alice2 ltfs[25412]: 6344 LTFS30252I Logical block protection is disabled.
Writing works with the default block size of 512KB. after removing the /etc/modprobe.d/sg.conf
file.
The file contained
options sg allow_dio=1
options sg def_reserved_size=1048576
These settings were not the default, but were added by me in the hope of increasing performance as suggested by Parameter settings of the sg driver. But apparently they are not compatible with this ATTO HBA.
So I'm closing this bug since it works with the default settings.
(Thanks a lot for your help, and sorry that the feedback promised for "some time next week" actually ended up taking 2 months)
@mivk,
Thank you very much for your all work. I updated the description about DIO setting in README.md.
After formatting and mounting tape, I can only write the first 524288 bytes block of the first file. Then the write fails.
ltfs was compiled with
./configure --enable-buggy-ifs
.Formatting works:
Mounting works (well, sometimes it doesn't, but maybe that should be a separate issue). Successful mount log:
But copying to the tape fails after 524288 bytes.
/var/log/ltfs.log
shows this: