search

dennwc / cas

Content Addressible Storage
Apache License 2.0
43 stars 3 forks source link

Workaround for filesystem without xattr ? #2

Open photocyte opened 1 year ago

photocyte commented 1 year ago

Hi there,

I'm working on a linux HPC cluster, that doesn't have xattr available. Is cas still able to function if xattrs are not available, or is there a workaround?

2023/02/21 10:14:51 xattr.FSet hisat2_12B1-RiboZero.merged.bam user.cas.size: operation not supported

Notably, the xattr setting at least does seem to work on macOS & the APFS filesystem:

user.cas.hash: sha256:b99ea5e9a6e80b9e3cd9f5df62bf6a6324ee79e6529384ae44099a92b630f58f
user.cas.mtime:
0000   5C 89 30 FF 0F E9 45 17                            ..0...E.

user.cas.size:
0000   19 C6 3B 65 00 00 00 00                            ..;e....
dennwc commented 1 year ago

Sorry, I never tested a setup where xattr is not available.

In theory it should work, but may require fixed to the error handling: it need to recognize "not supported" error and fallback to full hashing, or storing additional DB with hashes perhaps.

photocyte commented 1 year ago

Got it! A local sqlite DB in the .cas seems a workable solution?

Also, to the best of my knowledge and testing, xattrs are poorly supported on cloud storage systems, such as those described in https://rclone.org/overview/ .