search

LinearTapeFileSystem / ltfs

Reference implementation of the LTFS format Spec for stand alone tape drive
BSD 3-Clause "New" or "Revised" License
255 stars 76 forks source link

Tape Compression / mkltfs -c #434

Closed vguilleaume closed 7 months ago

vguilleaume commented 1 year ago

Dear community,

I would like to get your support to validate my understanding about the command mkltfs -c.

My objective is to make sure that the tape compression is not enable, then the tape content will not be compress. I do know that they are some options to do it, and going into OpenLTFS, i have seen this option, but never use it.

All the LTFS format task i did never include this option.

Could you help me to confirm that this option disable well the tape compression as i understand it ?

The situation as i have doubt about the effect of this option. Here is standard command i use to format a tape. I 'd like to understand if there is something - somewhere an indication about if the compression is set or not.

LTFS15000I   Starting mkltfs, LTFS version 2.4.4.1 (Prelim), log level 2."}
LTFS15041I Launched by 'mkltfs -f -d   /dev/sg2 -n MTC540L5'."}
LTFS15042I This binary is built for Linux   (x86_64)."}
LTFS15043I GCC version is 7.3.1 20180303   (Red Hat 7.3.1-5)."}
LTFS17087I Kernel version: Linux version   3.10.0-1160.92.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version   4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jun 20 11:48:01 UTC 2023   i386."}
LTFS17089I Distribution: CentOS Linux   release 7.9.2009 (Core)."}
LTFS17089I Distribution: NAME='CentOS   Linux'."}
LTFS17089I Distribution: CentOS Linux   release 7.9.2009 (Core)."}
LTFS17089I Distribution: CentOS Linux   release 7.9.2009 (Core)."}
LTFS15003I Formatting device   '/dev/sg2'."}
LTFS15004I LTFS volume blocksize:   524288."}
LTFS15005I Index partition placement   policy: None."}
"}
LTFS11337I Update index-dirty flag (1) -   NO_BARCODE (0x0x62d010)."}
LTFS17085I Plugin: Loading 'sg' tape   backend."}
LTFS30209I Opening a device through   sg-ibmtape driver (/dev/sg2)."}
LTFS30250I Opened the SCSI tape device   3.0.0.0 (/dev/sg2)."}
LTFS30207I Vendor ID is IBM     ."}
LTFS30208I Product ID is ULT3580-HH5     ."}
LTFS30214I Firmware revision is   F991."}
LTFS30215I Drive serial is   90WT013124."}
LTFS30285I The reserved buffer size of   /dev/sg2 is 524288."}
LTFS30294I Setting up timeout values from   RSOC."}
LTFS17160I Maximum device block size is   1048576."}
LTFS11330I Loading cartridge."}
LTFS30252I Logical block protection is   disabled."}
LTFS11332I Load successful."}
LTFS17157I Changing the drive setting to   write-anywhere mode."}
LTFS15049I Checking the medium   (load)."}
LTFS30252I Logical block protection is   disabled."}
LTFS15010I Creating data partition b on   SCSI partition 1."}
LTFS15011I Creating index partition a on   SCSI partition 0."}
LTFS17165I Resetting the medium's   capacity proportion."}
LTFS11097I Partitioning the   medium."}
LTFS30263I MODESELECT returns Mode   Parameters Rounded (-20101) /dev/sg2."}
LTFS11100I Writing label to partition   b."}
LTFS11278I Writing index to partition   b."}
LTFS30205I READ_ATTR (0x8c) returns   -20501."}
LTFS30263I READ_ATTR returns Invalid   Field in CDB (-20501) /dev/sg2."}
LTFS30233I Cannot read attribute   (-20501)."}
LTFS11336I The attribute does not exist.   Ignore the expected error."}
LTFS17235I Writing index of NO_BARCODE to   b (Reason: Format, 0 files) 90WT013124."}
LTFS17236I Wrote index of NO_BARCODE (Gen   = 1, Part = b, Pos = 5, 90WT013124)."}
LTFS11337I Update index-dirty flag (0) -   NO_BARCODE (0x0x62d010)."}
LTFS11100I Writing label to partition   a."}
LTFS11278I Writing index to partition   a."}
LTFS30205I READ_ATTR (0x8c) returns   -20501."}
LTFS30263I READ_ATTR returns Invalid   Field in CDB (-20501) /dev/sg2."}
LTFS30233I Cannot read attribute   (-20501)."}
LTFS11336I The attribute does not exist.   Ignore the expected error."}
LTFS17235I Writing index of NO_BARCODE to   a (Reason: Format, 0 files) 90WT013124."}
LTFS17236I Wrote index of NO_BARCODE (Gen   = 1, Part = a, Pos = 5, 90WT013124)."}
LTFS15013I Volume UUID is:   14c76095-641a-4014-add5-ce59595a88c2."}
"}
LTFS15019I Volume capacity is 1425   GB."}
LTFS30252I Logical block protection is   disabled."}
LTFS15024I Medium formatted   successfully."}

Thanks Valery

piste-jp commented 1 year ago

Compression setting on tape drive is not important especially LTO and IBM enterprise tape drive. It means weakness of data on tape is not changed by compression setting because of internal data structure on tape. So we never recommend that user uses LTFS with compression off.

To enable compression, you can use space which is generated by compressed data as a buffer. So I strongly recommend that compression on is used all time.

You can confirm this option really disable compression by writing a 0-fill large file. Because 0-fill data is maximumly compressed (may be 20:1?). On a compression on tape, used capacity is not so increased. But on a no-comprerssion on tape, it is increased almost same as file size.

vguilleaume commented 1 year ago

thanks @piste-jp-ibm for this update and advice. I understand your idea about compression - non compression. I'm personally used to let it activated as you mentioned.

My question is related to a very special case, where we do have a request for data stored to LTFS tape without compression. I'm looking for all available options where we can provide evidence that no compression is active when writing to the tapes.

I don't believe that the customer will be fine with the option about the 0-fill large file. He has strong requirement about how data are stored on the tape itself (LTFS format) and also about the compression that has to be disable

I'm evaluating the option to disable the compression on the tape drive itself, based on your knowledge is this redundant or complementary with the ltfs -c option

piste-jp commented 1 year ago

My question is related to a very special case, where we do have a request for data stored to LTFS tape without compression. I'm looking for all available options where we can provide evidence that no compression is active when writing to the tapes.

I'm not sure how much you know about compression of tape/tape drive. But actually, tape drive has a H/W on-the-fly compression circuit in it and compression is done without any cost. The switch is in the drive itself and user need to specify the compression setting before writing data by MODE_SELECT command, 'compression ON' is default.

I'm not sure what you want to do from the description above, but if you want to switch the compression setting by 'file', it is a terrible idea for LTFS. Because we have a chance to write 2 files (or more) at a same time. If one stream 'file' shall be compressed and the other shall not be compressed, LTFS might switch compression setting frequently and it might cause dramatic performance degradation.

At this time, compression setting can be specified only at format time. The setting is stored into the LTFS label. When LTFS mount a tape, it reads the LTFS labels and make compression setting.

I don't believe that the customer will be fine with the option about the 0-fill large file. He has strong requirement about how data are stored on the tape itself (LTFS format) and also about the compression that has to be disable I'm evaluating the option to disable the compression on the tape drive itself, based on your knowledge is this redundant or complementary with the ltfs -c option

I'm not sure again what you want to do. Need to write a highly compressible data if you want to check the code and drive work well.

If you just want to indicate the current compression setting that LTFS detected, I think you can use ltfs.volumeCompression VEA described into https://www.ibm.com/docs/en/spectrum-archive-sde/2.4.5?topic=attributes-virtual-extended.

The code is below. You might be able to get true or false.

https://github.com/LinearTapeFileSystem/ltfs/blob/18ae70c3caae96caad687c18f763b719c3c0c31e/src/libltfs/xattr.c#L722-L724.

vguilleaume commented 1 year ago

Thanks for this comprehensive update,

From your detailed explanation, i made several tests today, in working on the drive itself.

I executed a mt command to get the tape drive off.

Basically: mt -f /dev/nstx compression 0 and mt -f /dev/nstx defcompression 0

When doing this i can verify if the tape compression is enable or not (in mounting a tape in the drive:

# tapeinfo -f /dev/stx
Product Type: Tape Drive
Vendor ID: 'IBM     '
Product ID: 'ULT3580-HH5     '
Revision: 'F991'
Attached Changer API: No
SerialNumber: '90WT013124'
MinBlock: 1
MaxBlock: 8388608
SCSI ID: 0
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: 0x58
Density Code: 0x58
BlockSize: 0
DataCompEnabled: no
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: -1
Partition 0 Size in Kbytes: -1
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 1
MaxPartitions: 1
Partition0: 38
Partition1: 1444

I experimented some strange behavior:

And last but not least, the most important: after the LTFS mounted or even after i did the LTFS tape format, i have can see that the DataCompEnabled setting went back to on, meaning that the compression is enable again.

I wrote data on the tape, i can see that compression is still also active (i retrieved the real capacity stored through log sense and the compression ratio too).

With your experience on tape drive, compression, and LTFS, could we imagine that the compression is back on due to LTFS usage?

And then, can I go to the conclusion that we have to use the mt command to the compression at off and then format the tape with mkltfs -c in specifying the compression as off too ?

Thanks again for your support and experience in this area

piste-jp commented 12 months ago

As I said in my previous comment. Compression setting of LTFS formatted tape is written on the tape as a part of the LTFS label. The LTFS label is read in the middle of mount process and LTFS set the compression setting.

Another implicit rule is tape application never restore drive setting at start time at all. In other words, LTFS never revert the compression setting to off at unmount even if it is set before mount process.

So your description below is quite natural to me.

And last but not least, the most important: after the LTFS mounted or even after i did the LTFS tape format, i have can see that the DataCompEnabled setting went back to on, meaning that the compression is enable again.

piste-jp commented 12 months ago

And then, can I go to the conclusion that we have to use the mt command to the compression at off and then format the tape with mkltfs -c in specifying the compression as off too ?

No, just issue mkltfs -c. Previous compression setting is not related at all. mkltfs shall set the compression off and write the compression-off setting LTFS label on tapeat format. And ltfs shall read the compression-off setting LTFS label and set teh drive compression off.

vguilleaume commented 12 months ago

Thanks for this update, you confirm what i had in mind.

We will then proceed with mkltfs -c to disable the compression, we will include this into all LTFS format commands . And we can conclude that we will not rely on the mt commands anymore.

piste-jp commented 7 months ago

Closing because there is no update for long time.