search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
605 stars 218 forks source link

AzCopy fails to write xattr #2761

Open tsvietOK opened 2 months ago

tsvietOK commented 2 months ago

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.25.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux (Ubuntu 22.04 VM in Azure)

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy sync "https://\.blob.core.windows.net/\/\/?\" "." --delete-destination true --compare-hash "MD5" --cap-mbps 400

What problem was encountered?

When trying to sync files from storage account to Ubuntu 22.04 I getting xattr related errors:

2024/07/24 16:15:38 failed to write hash data for edd - Copy (3).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/24 16:15:38 failed to write hash data for edd - Copy.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/24 16:15:38 failed to write hash data for edd - Copy (2).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode

Important! Even when it shows errors in the log and shows all operations as failed, it downloads files anyway.

How can we reproduce the problem in the simplest way?

  1. Upload test files to storage account
  2. Create Ubuntu 22.04 VM in Azure
  3. Attach data disk
  4. Mount data disk (as ext4)
  5. Sync files from storage account to this data disk using the command azcopy sync (as above)

    Have you found a mitigation/solution?

    I installed xattr package, but it didn't help. Tried to use OS agnostic way, but there is another issue with it #2758 The potential reason for this issue is that you are trying to write xattr without "user." prefix. I tried to set xattrs for test file by command xattr -w sss testVal test.xml and it failed with error [Errno 95] Operation not supported: b'test.xml', but it works as xattr -w user.sss testVal test.xml. As far as I understand, user_xattr is enabled for ext4 filesystem by default. Also, I tried to enable it explicitly (via /etc/fstab). It is described here: https://stackoverflow.com/questions/60893372/why-must-a-user-be-prepended-to-the-name-when-setting-a-files-xattr-with-os

tsvietOK commented 2 months ago

Command log:

INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job d0c17a5c-fb5d-c449-503e-38a02477af0f has started
Log file is located at: /home/user/.azcopy/d0c17a5c-fb5d-c449-503e-38a02477af0f.log

INFO: One or more hash storage operations (read/write) have failed. Check the scanning log for details.
INFO: Deleting extra file: edd - Copy (7).txt
INFO: Deleting extra file: edd.txt
INFO: Deleting extra file: edd - Copy (11).txt
INFO: Deleting extra file: edd - Copy (10).txt
INFO: Deleting extra file: edd - Copy (5).txt
INFO: Deleting extra file: edd - Copy (4).txt
INFO: Deleting extra file: 123/ssss.txt
INFO: Deleting extra file: edd - Copy (2).txt
INFO: Deleting extra file: edd - Copy (8).txt
INFO: Deleting extra file: edd - Copy (6).txt
INFO: Deleting extra file: edd - Copy (9).txt
INFO: Deleting extra file: edd - Copy.txt
INFO: Deleting extra file: edd - Copy (3).txt
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
0.0 %, 0 Done, 3 Failed, 0 Pending, 3 Total, 2-sec Throughput (Mb/s): 268.3674

Job d0c17a5c-fb5d-c449-503e-38a02477af0f Summary
Files Scanned at Source: 3
Files Scanned at Destination: 13
Elapsed Time (Minutes): 0.1002
Number of Copy Transfers for Files: 3
Number of Copy Transfers for Folder Properties: 0
Total Number of Copy Transfers: 3
Number of Copy Transfers Completed: 0
Number of Copy Transfers Failed: 3
Number of Deletions at Destination: 13
Total Number of Bytes Transferred: 0
Total Number of Bytes Enumerated: 270993275
Final Job Status: Failed

Scanning log:

2024/07/25 17:35:56 AzcopyVersion  10.25.1
2024/07/25 17:35:56 OS-Environment  linux
2024/07/25 17:35:56 OS-Architecture  amd64
2024/07/25 17:35:56 Log times are in UTC. Local time is 25 Jul 2024 17:35:56
2024/07/25 17:35:56 failed to read hash data for edd - Copy (11).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (3).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (2).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (7).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (5).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (10).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (4).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (8).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (6).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (9).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for 123/ssss.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (11).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (2).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (5).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (10).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (7).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (3).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (8).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (6).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (9).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (4).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for 123/ssss.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 ==> REQUEST/RESPONSE (Try=1/12.696453ms, OpTime=54.29216ms) -- RESPONSE SUCCESSFULLY RECEIVED
   HEAD https://<storage>.blob.core.windows.net/<container>/<folder>?se=2024-07-26T01%3A35%3A41Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2024-07-25T17%3A35%3A41Z&sv=2022-11-02
   X-Ms-Request-Id: [fec348af-301e-0037-0fb9-de42b6000000]

2024/07/25 17:35:56 ==> REQUEST/RESPONSE (Try=1/6.572306ms, OpTime=6.613234ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://<storage>.blob.core.windows.net/<container>?comp=list&delimiter=%2F&include=metadata&prefix=<prefix>%2F&restype=container&se=2024-07-26T01%3A35%3A41Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2024-07-25T17%3A35%3A41Z&sv=2022-11-02
   X-Ms-Request-Id: [fec348ba-301e-0037-17b9-de42b6000000]

2024/07/25 17:35:56 Deleting extra file: edd - Copy (7).txt
2024/07/25 17:35:56 Deleting extra file: edd.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (11).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (10).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (5).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (4).txt
2024/07/25 17:35:56 Deleting extra file: 123/ssss.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (2).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (8).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (6).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (9).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (3).txt
2024/07/25 17:36:02 Closing Log
tsvietOK commented 2 months ago

Installed xattr: $ dpkg -s xattr

Package: xattr
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 17
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: python-xattr
Version: 0.9.7-1build4
Depends: python3:any, python3-xattr (= 0.9.7-1build4), python3-pkg-resources
Description: tool for manipulating filesystem extended attributes
 xattr tool allows manipulation of the filesystem extended attributes present
 in some operating systems (GNU/Linux included).
 .
 This package contains the xattr cli tool.
Homepage: https://github.com/xattr/xattr
Original-Maintainer: Federico Ceratto <federico@debian.org>
tsvietOK commented 2 months ago

https://go.googlesource.com/sys.git/+/master/unix/xattr_bsd.go#31

func xattrnamespace(fullattr string) (ns int, attr string, err error) {
  s := strings.IndexByte(fullattr, '.')
  if s == -1 {
    return -1, "", ENOATTR
  }
   
  namespace := fullattr[0:s]
  attr = fullattr[s+1:]
   
  switch namespace {
    case "user":
      return EXTATTR_NAMESPACE_USER, attr, nil
    case "system":
      return EXTATTR_NAMESPACE_SYSTEM, attr, nil
    default:
      return -1, "", ENOATTR
  }
}
tsvietOK commented 2 months ago

Fix suggested: https://github.com/Azure/azure-storage-azcopy/pull/2762

adreed-msft commented 1 month ago

Hi there. Apologies for the delayed response. Xattr has to be available at a filesystem level, which is usually a flag when mounting the fs.

There are two potential mitigations here

1) Enable xattr on the target filesystem 2) Use a different hash storage mode (check the docs for --local-hash-storage-mode and --hash-meta-dir for a dependency agnostic solution)

tsvietOK commented 1 month ago

As far as I understand, user_xattr is enabled for ext4 filesystem by default.

OS agnostic solution also has its own issue as I described in issue description.

tsvietOK commented 1 month ago

image

tsvietOK commented 1 month ago

@adreed-msft I tested my changes from PR #2762 and it works as expected.

tsvietOK commented 1 month ago

More information about ENOTSUP: https://manpages.debian.org/unstable/manpages-dev/setxattr.2.en.html#EDQUOT image