ltfs_ordered_copy is really slow

[rezo552]

Hi,

Im still trying to understand what is the best way to retrieve files in batch from LTFS. The ltfs_ordered_copy takes ages to retrieve couple of hundreds of megs and it seems the drive is constantly seeking. Are HP drives supported at all? Thanks

Z.

After 40 minutes, I was only able to retrieve 17 files (cca. 20 megs each)

root@pop-os:/home/rezo# ./ltfs_ordered_copy /ltfs/n64/ /home/rezo/ -r
Tape order aware copy for LTFS
Destination /home/rezo/ is not LTFS
On disk sources: 1058
Source tapes: 0
Copying on 1058 disk files with 8 threads
File copy from disk is on going: 17/1058^C^Y^Z

[piste2750]

I believe this doesn't depend on the drive but depends on the version of LTFS.

Which LTFS version are you using on /ltfs?

From the log of ltfs_ordered_copy, it doesn't detect specified files is not on the LTFS.

On disk sources: 1058
Source tapes: 0
Copying on 1058 disk files with 8 threads
File copy from disk is on going: 17/1058

The LTFS you are using needs to return a correct data against VEAs, ltfs.volumeUUID, ltfs.partition and ltfs.startblock. (In Linux, user. prefix is required like user.ltfs.volumeUUID)

Once ltfs_ordered_copy capture those info, it tries to reorder the source files by ltfs.startblock. As a result, seek might be minimized.

The code below is your reference.

https://github.com/LinearTapeFileSystem/ltfs/blob/42c08710bda914e2bf0ac3749c69d830e9b45577/src/utils/ltfs_ordered_copy#L61-L78

[rezo552]

Im using this version: rezo@pop-os:~$ ltfs -V LTFS version 2.5.0.0 (Prelim). LTFS Format Specification version 2.4.0.

Does it matter if the drive was NOT written with ltfs_order_copy?

[rezo552]

If Im copying to the tape it seems to detect LTFS

Tape order aware copy for LTFS Destination /ltfs is LTFS On disk sources: 15082 Source tapes: 0 Copying on 15082 disk files with 1 threads File copy from disk is on going: 630/15082

[piste2750]

I would like to check the LTFS returns VEA correctly.

Could you try the commands below against a source file ?

Linux:

attr -g ltfs.volumeUUID [srcfile]
attr -g ltfs.partition [srcfile]
attr -g ltfs.ltfs.startblock [srcfile]

Mac:

xattr -p ltfs.volumeUUID [srcfile]
xattr -p ltfs.partition [srcfile]
xattr -p ltfs.ltfs.startblock [srcfile]

[rezo552]

now I got it working

root@pop-os:/home/rezo# find /ltfs -type f | ./ltfs_ordered_copy -t /smbnasusb --keep-tree=/ltfs Tape order aware copy for LTFS Destination /smbnasusb is not LTFS

On disk sources: 0 Source tapes: 1 Processing tape b2ce8374-7e34-41a7-bf1a-818171a69993, Partition b Sorting 15083 files on b2ce8374-7e34-41a7-bf1a-818171a69993 (b) Copying files on b2ce8374-7e34-41a7-bf1a-818171a69993 (b) File copy from tape is on going: 15083/15083 Copied 15083 files.

diff.txt

The issue was with the xattr python library, some functions are renamed

[piste2750]

Thank you for your investigation. And it is interesting...

I confirmed the functions you pointed out was obsoleting from v0.4. So it is a time to with to new functions as you fixed. I will do that in the near future.

[Nooby1]

for me on python 3.9, without this change the script just breaks and will not run

python3-xattr is already the newest version (0.9.7-1+b1).

module "xattr" has no attribute "get"

[piste2750]

Thank you for your investigation. And it is interesting...

I confirmed the functions you pointed out was obsoleting from v0.4. So it is a time to with to new functions as you fixed. I will do that in the near future.

@rezo552 , @Nooby1

I tried to work this today but I realize something funny. From the document of pyxattr module, getxattr() and setxattr() is obsoleting from v0.4.. -> See https://pyxattr.k1024.org/module.html#module-xattr

The ltfs_ordered_copy requires the pyxattr module and it is described here.

Could you confirm you are using the pyxattr module? I would like to note that there is another module xattr to handle extended attribute.

• pyxattr module -> https://pypi.org/project/pyxattr/
• xattr module -> https://pypi.org/project/xattr/

[Nooby1]

I was using the xattr module, not pyxattr, the script seems to work with some small changes with xattr however

[ia2115]

can you provide the edit verion of the script please? on freebsd its just broken nomatter what i try :/