search

LinearTapeFileSystem / ltfs

Reference implementation of the LTFS format Spec for stand alone tape drive
BSD 3-Clause "New" or "Revised" License
246 stars 73 forks source link

Support Quantum LTO drives (as quantum source is offline) #219

Closed z3cko closed 3 years ago

z3cko commented 3 years ago

Is your feature request related to a problem? Please describe.

Quantum's LTFS Linux files are offline since over a month. See https://www.quantum.com/ServiceandSupport/OpenSource/LTFS/Index.aspx

Describe the solution you'd like It would be ideal if quantum drives would also be supported in this repo (which they are currently not). I guess it would be just a matter of adding the proper device IDs. can you clarify?

Describe alternatives you've considered I tried getting this version of ltfs to run with archlinux, but it does not work yet. https://aur.archlinux.org/packages/ltfs/ https://aur.archlinux.org/packages/ltfs-quantum/

Additional context Add any other context or screenshots about the feature request here.

piste-jp commented 3 years ago

I guess it would be just a matter of adding the proper device IDs. can you clarify?

No, it is NOT just add the vendor/device IDs. Now I luckely had a support from HP's LTFS team and they kindly contribute to us. So HP drives are now supported. But code change is not so simple. You can see the source (https://github.com/LinearTapeFileSystem/ltfs/blob/master/src/tape_drivers/linux/sg/sg_tape.c), we have some conditionals like priv->vendor == VENDOR_HP.

Major consideration points are

At this time, I added the definitions of Quantum's HH drives into the list in the code (See #209). Now the code is using IBM LTO drive logic if Quantum HH drive is found. It is lucky if it works fine. And I assume so from (https://github.com/royalmoose/ltfs-quantum). But I can't confirm this because I don't have any Quantum drives.

For Quantum's FH drives, I have no idea how to support this. I will add to support of them in the future but this is not a priority for me at this time. I hope someone who has there drives contributes the code.

z3cko commented 3 years ago

Thank you very much for this answer. I have access to a quantum HH drive and I will be able to test this on the weekend. Regarding https://github.com/royalmoose/ltfs-quantum -- is there some more info on the fork? Or how do you assume that it works? (i seem to not be able to find a lot of information out there on the actual adaptations of this fork repo).

dericed commented 3 years ago

Hi, I have a QUANTUM ULTRIUM-HH8 and am testing with the master branch here. So far I can seemingly format a tape with mkltfs:

≈:≈ mkltfs --device=0 -f 
LTFS15000I Starting mkltfs, LTFS version 2.4.2.1 (Prelim), log level 2.
LTFS15041I Launched by "mkltfs --device=0 -f".
LTFS15042I This binary is built for Mac OS X .
LTFS15043I GCC version is 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4).
LTFS17087I Kernel version: Darwin Kernel Version 18.7.0: Thu Jun 18 20:50:10 PDT 2020; root:xnu-4903.278.43~1/RELEASE_X86_64.
LTFS15003I Formatting device '0'.
LTFS15004I LTFS volume blocksize: 524288.
LTFS15005I Index partition placement policy: None.

LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x7f8f35505460).
LTFS17085I Plugin: Loading "iokit" tape backend.
LTFS30810I Opening a device through iokit driver (0).
LTFS30814I Vendor ID is QUANTUM .
LTFS30815I Product ID is 'ULTRIUM-HH8     '.
LTFS30816I Firmware revision is M571.
LTFS30817I Drive serial is 1097001104.
LTFS17160I Maximum device block size is 1048576.
LTFS11330I Loading cartridge.
LTFS11332I Load successful.
LTFS17157I Changing the drive setting to write-anywhere mode.
LTFS15049I Checking the medium (load).
LTFS15010I Creating data partition b on SCSI partition 1.
LTFS15011I Creating index partition a on SCSI partition 0.
LTFS17165I Resetting the medium's capacity proportion.
LTFS11097I Partitioning the medium.
LTFS30865I MODESELECT returns Mode Parameters Rounded (-20101) 0.
LTFS11100I Writing label to partition b.
LTFS11278I Writing index to partition b.
LTFS30808I READ_ATTR (0x8c) returns -20501.
LTFS30865I READ_ATTR returns Invalid Field in CDB (-20501) 0.
LTFS30836I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of NO_BARCODE to b (Reason: Format, 0 files) 1097001104.
LTFS17236I Wrote index of NO_BARCODE (b, 1097001104).
LTFS11337I Update index-dirty flag (0) - NO_BARCODE (0x0x7f8f35505460).
LTFS11100I Writing label to partition a.
LTFS11278I Writing index to partition a.
LTFS30808I READ_ATTR (0x8c) returns -20501.
LTFS30865I READ_ATTR returns Invalid Field in CDB (-20501) 0.
LTFS30836I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of NO_BARCODE to a (Reason: Format, 0 files) 1097001104.
LTFS17236I Wrote index of NO_BARCODE (a, 1097001104).
LTFS15013I Volume UUID is: ef0e6ee3-ee36-46e5-a02c-6256387cca97.

LTFS15019I Volume capacity is 5732 GB.
LTFS30854I Logical block protection is disabled.
LTFS15024I Medium formatted successfully.

However, I haven't been able to mount the tape:

≈:≈ ltfs
307 LTFS14000I LTFS starting, LTFS version 2.4.2.1 (Prelim), log level 2.
307 LTFS14058I LTFS Format Specification version 2.4.0.
307 LTFS14104I Launched by "ltfs".
307 LTFS14105I This binary is built for Mac OS X .
307 LTFS14106I GCC version is 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4).
307 LTFS17087I Kernel version: Darwin Kernel Version 18.7.0: Thu Jun 18 20:50:10 PDT 2020; root:xnu-4903.278.43~1/RELEASE_X86_64.
307 LTFS14063I Sync type is "time", Sync time is 300 sec.
307 LTFS17085I Plugin: Loading "iokit" tape backend.
307 LTFS17085I Plugin: Loading "unified" iosched backend.
307 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
307 LTFS30810I Opening a device through iokit driver (0).
307 LTFS30814I Vendor ID is QUANTUM .
307 LTFS30815I Product ID is 'ULTRIUM-HH8     '.
307 LTFS30816I Firmware revision is M571.
307 LTFS30817I Drive serial is 1097001104.
307 LTFS17160I Maximum device block size is 1048576.
307 LTFS11330I Loading cartridge.
307 LTFS11332I Load successful.
307 LTFS17157I Changing the drive setting to write-anywhere mode.
307 LTFS11005I Mounting the volume.
307 LTFS30819W A length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523291).
307 LTFS30819W A length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523291).
307 LTFS12049E Cannot read: backend call failed (-21716).
307 LTFS17039E XML parser: failed to read a block from the medium (-21716).
307 LTFS17037E XML parser: failed to read from XML stream.
307 LTFS17016E Cannot parse index direct from medium.
307 LTFS11194W Cannot read index: failed to read and parse XML data (-1014).
307 LTFS11024E Cannot mount volume: read index failed on the index partition.
307 LTFS14013E Cannot mount the volume.
307 LTFS30854I Logical block protection is disabled.

This is a lot further than I could get before though, so I wanted to thank you for your work on https://github.com/LinearTapeFileSystem/ltfs/pull/209 and offer any testing that I could do.

piste-jp commented 3 years ago

Regarding https://github.com/royalmoose/ltfs-quantum -- is there some more info on the fork?

I think this fork said Quantum HH drives probably work with the same logic of IBM's one. This is the most important information to me.

piste-jp commented 3 years ago

However, I haven't been able to mount the tape:

@dericed,

I think your HBA is little bit tricky. Can you provide information of your HBA? I think this HBA doesn't handle the residual length in SCSI layer correctly.

This issue is already reported by #51 and resolved by #52. May be LTFS can handle this case correctly if you specify --enable-buggy-ifs to the configure script and rebuild the code.

dericed commented 3 years ago

Hi @piste-jp-ibm, thanks so much. I've tested formating, mounting and reading/writing some small sets of files and it appears to be behaving as I've been accustomed to. Please let me know if I can provide any other helpful info or run tests while I have access to this setup.

Here's information on my HBA:

system_profiler SPSASDataType
SAS:

    SAS Domain 0:

      Vendor: HighPoint Technologies, Inc.
      Product: RocketRAID 2722 SAS Controller
      Revision: 4.00
      Initiator Identifier: 255
      SAS Address: 50:01:93:C0:11:03:00:00

        SCSI Target Device @ 32:

          SAS Address: 50:01:93:C0:11:03:40:20
          SCSI Target Identifier: 32
          SCSI Peripheral Device Type: 3
          Manufacturer: HPT
          Model: RCM DEVICE
          Revision: 4.00

            SCSI Logical Unit @ 0:

              LUN Address: 00:00:00:00:00:00:00:00

        SCSI Target Device @ 0:

          SAS Address: 50:05:07:63:12:57:28:04
          SCSI Target Identifier: 0
          SCSI Peripheral Device Type: 1
          Manufacturer: QUANTUM
          Model: ULTRIUM-HH8
          Revision: M571

            SCSI Logical Unit @ 0:

              LUN Address: 00:00:00:00:00:00:00:00
z3cko commented 3 years ago

Hi @piste-jp-ibm, thanks so much. I've tested formating, mounting and reading/writing some small sets of files and it appears to be behaving as I've been accustomed to. Please let me know if I can provide any other helpful info or run tests while I have access to this setup.

hi @dericed - please keep us posted if you experience something fishy. this is important info for all quantum owners and users i guess.

@piste-jp-ibm - can we add quantum to the supported drives list just yet?

piste-jp commented 3 years ago

@dericed

Thank you for your info. I created the page provides which HBA need to specify --enable-buggy-ifs on Wiki.

@piste-jp-ibm - can we add quantum to the supported drives list just yet?

@z3cko , Yha, it seems to be time to add Quantum HH drives in the list. I will modify the drive list on README.md. Or you can make a PR. I can merge it.

dericed commented 3 years ago

Thanks all. Is there a reason that --enable-buggy-ifs is not the default behavior or any reason to avoid it when not needed? I've been using the installer at https://github.com/piste2750/homebrew-ltfs/blob/master/ltfs.rb to support keep ltfs up-to-date and could make a PR to add this option to the installer, but am uncertain under what cases it should not be used.

piste-jp commented 3 years ago

Is there a reason that --enable-buggy-ifs is not the default behavior or any reason to avoid it when not needed?

The reason why I want to make it optional is to have confidence about data accuracy from the hardware. In this case, the drive reports the record length correctly in the SCSI command layer. But HBA reports wrong length in the SCSI transport layer.

For work around, I trust the drive's report when --enable-buggy-ifs is enabled. But this logic have a chance to return wrong data to the caller if they are using buggy HBA. (I think RocketRAID is not so bad. But I saw another buggy HBA in the past...)

So I want to make it optional to rise an alert to the users before using a buggy HBA.

Historically, tape support on HBA is not a priority. So I saw many HBAs that doesn't support longer payload or another required features for tape. Transfer length support is one of them.

piste-jp commented 3 years ago

Added the Quantum drives into the support list in #220.

xosevp commented 3 years ago

Quantum's LTFS Linux files are offline since over a month. [...] It would be ideal if quantum drives would also be supported in this repo (which they are currently not). I guess it would be just a matter of adding the proper device IDs. can you clarify?

Current Quantum fork is in the v2.4.0.2-Quantum branch at: https://github.com/royalmoose/ltfs-quantum/tree/v2.4.0.2-Quantum The code to support Quantum drives is added in this commit: https://github.com/royalmoose/ltfs-quantum/commit/0a41225db984035e2927d07554b7a5f87bbc01f2 And there are a lot of exceptions in the code, look for QUANTUM_BUILD

Firmware and SAS Controllers compatibility is described in: http://qsupport.quantum.com/freedownloads/ltfs/0-13664-01_RevH.pdf