Push to RIA storage-only remote from mac to linux fails

ddeepwell commented 1 year ago

What is the problem?

Pushing data to a storage-only RIA remote fails when the remote is linux and the local repo is on a mac.

What steps will reproduce the problem?

datalad create test_to_storage_only_ria
cd test_to_storage_only_ria
head -c 1000 /dev/urandom > dummy.bin
datalad save -m "Add annex file"
datalad create-sibling-ria -s arc --alias test --shared group --group rsws --storage-sibling only --new-store-ok ria+ssh://arc.ucalgary.ca:/path/to/storage
datalad push --to arc

Here's the error message:

CommandError: 'git -c diff.ignoreSubmodules=none annex copy --batch -z --to arc --fast --json --json-error-messages --json-progress -c annex.dotfiles=true' failed with exitcode 1 under /private/tmp/test_storage_only_ria [info keys: stdout_json] > to arc... Unable to remove /path/to/storage/897/03ee8-379e-4f75-ad4c-41ab8271650c/ora-remote-cc3d0f15-88df-4c1e-b26a-8ba3f0634190/transfer/MD5E-s1000--9fa75b5852530a249e1b83c6e0e7d21d.bin or to obtain write permission in parent directory. -caused by- stat -f%Dp /path/to/storage/897/03ee8-379e-4f75-ad4c-41ab8271650c/ora-remote-cc3d0f15-88df-4c1e-b26a-8ba3f0634190/transfer failed: This could have failed because --fast is enabled.

DataLad information

datalad version 0.18.3 python 3.10.8

system

distribution: darwin/22.3.0 13.2.1/arm64
encoding:
- default: utf-8
- filesystem: utf-8
- locale.prefered: UTF-8
max_path_length: 282
name: Darwin
release: 22.3.0
type: posix
version: Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103

Additional context

May be related to datalad/datalad-ria#39

Note that the remote is linux while the local machine is macOS. The error message shows that the stat command fails. Running stat -f%Dp file.txt on the local (mac) works, but fails on the remote (linux). Could the wrong command be sent to the remote? Running the steps listed above on linux to a linux remote works.

Have you had any success using DataLad before?

Becoming fairly proficient, having used it steadily over the last year

yarikoptic commented 1 year ago

hm, doesn't it look like https://github.com/datalad/datalad/issues/7296 which as mysteriously came up to our CI runs and then was gone?

ddeepwell commented 1 year ago

Based on the error message from datalad/datalad#7296, it does look similar, but for different reasons. In that issue, it looked like rm was the issue, whereas it is stat here.

ddeepwell commented 1 year ago

Just in case it wasn't git-annex, I upgraded my somewhat old version of git-annex (8.20210631-g9905ec19a) to the latest (10.20230329), to no effect.

bpoldrack commented 1 year ago

Could the wrong command be sent to the remote?

Yes, I suppose that's the issue. Remote execution needs its own detection what system it is running on.

jsheunis commented 1 year ago

I can confirm that this issue reproduces for me with the following setup:

Dataset: macOS Big Sur 11.7.6
RIA sibling: Linux server (juseless)

$ datalad create-sibling-ria -s arc --alias test --shared group --group rsws --storage-sibling only --new-store-ok ria+ssh://juseless.inm7.de:/<path-to-ria-location>

[INFO   ] create storage sibling 'arc' ...
create-sibling-ria(ok): /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria (dataset)

$ datalad push --to arc

Transfer data to 'arc':  50%|████████████████████████████████████████████▌                                            | 2.00/4.00 [00:00<00:00, 5.88k Steps/s
CommandError: 'git -c diff.ignoreSubmodules=none annex copy --batch -z --to arc --fast --json --json-error-messages --json-progress -c annex.dotfiles=true' failed with exitcode 1 under /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria [info keys: stdout_json]
> to arc...
  Unable to remove /<path-to-ria-location>/345/4b3bc-8bb6-4de5-9ed4-6b516031d14f/ora-remote-a22c7b35-11fd-498d-b4f6-cb14b0ffbe52/transfer/MD5E-s1000--fc8e14ed76e9387f7408ef8a71cb2aca.bin or to obtain write permission in parent directory. -caused by- stat -f%Dp /<path-to-ria-location>/345/4b3bc-8bb6-4de5-9ed4-6b516031d14f/ora-remote-a22c7b35-11fd-498d-b4f6-cb14b0ffbe52/transfer failed:
  This could have failed because --fast is enabled.
copy: 1 failed

Extra info:

datalad wtf

``` # WTF ## configuration ## datalad - version: 0.18.3+25.gc7d31e1d8 ## dataset - branches: - git-annex@c391781 - main@3704a85 - id: 3454b3bc-8bb6-4de5-9ed4-6b516031d14f - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria - repo: AnnexRepo ## dependencies - annexremote: 1.6.0 - boto: 2.49.0 - cmd:7z: 17.04 - cmd:annex: 10.20230227 - cmd:bundled-git: UNKNOWN - cmd:git: 2.31.1 - cmd:ssh: 8.1p1 - cmd:system-git: 2.31.1 - cmd:system-ssh: 8.1p1 - humanize: 4.6.0 - iso8601: 1.1.0 - keyring: 23.13.1 - keyrings.alt: 4.2.0 - msgpack: 1.0.5 - platformdirs: 3.5.0 - requests: 2.30.0 ## environment - LANG: en_US.UTF-8 - LANGUAGE: en_US.UTF-8 - LC_ALL: en_US.UTF-8 - LC_CTYPE: en_US.UTF-8 - LC_TERMINAL: iTerm2 - LC_TERMINAL_VERSION: 3.4.5 - PATH: /Users/jsheunis/.rbenv/shims:/Users/jsheunis/opt/miniconda3/envs/next2/bin:/Users/jsheunis/opt/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/jsheunis/.dotfiles/bin:/usr/local/sbin:/usr/X11R6/bin:/usr/local/games:/usr/games ## extensions - next: - description: What is next in DataLad - entrypoints: - datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV: - class: CreateSiblingWebDAV - load_error: None - module: datalad_next.commands.create_sibling_webdav - names: - create-sibling-webdav - datalad_next.commands.credentials.Credentials: - class: Credentials - load_error: None - module: datalad_next.commands.credentials - names: - datalad_next.commands.download.Download: - class: Download - load_error: None - module: datalad_next.commands.download - names: - download - datalad_next.commands.tree.TreeCommand: - class: TreeCommand - load_error: None - module: datalad_next.commands.tree - names: - tree - load_error: None - module: datalad_next - version: 1.0.0b2+64.g9a68f85 ## git-annex - build flags: - Assistant - Webapp - Pairing - FsEvents - TorrentParser - MagicMime - Benchmark - Feeds - Testsuite - S3 - WebDAV - dependency versions: - aws-0.24 - bloomfilter-2.0.1.0 - cryptonite-0.30 - DAV-1.3.4 - feed-1.3.2.1 - ghc-9.4.4 - http-client-0.7.13.1 - persistent-sqlite-2.13.1.1 - torrent-10000.1.1 - uuid-1.3.15 - yesod-1.6.2.1 - key/value backends: - SHA256E - SHA256 - SHA512E - SHA512 - SHA224E - SHA224 - SHA384E - SHA384 - SHA3_256E - SHA3_256 - SHA3_512E - SHA3_512 - SHA3_224E - SHA3_224 - SHA3_384E - SHA3_384 - SKEIN256E - SKEIN256 - SKEIN512E - SKEIN512 - BLAKE2B256E - BLAKE2B256 - BLAKE2B512E - BLAKE2B512 - BLAKE2B160E - BLAKE2B160 - BLAKE2B224E - BLAKE2B224 - BLAKE2B384E - BLAKE2B384 - BLAKE2BP512E - BLAKE2BP512 - BLAKE2S256E - BLAKE2S256 - BLAKE2S160E - BLAKE2S160 - BLAKE2S224E - BLAKE2S224 - BLAKE2SP256E - BLAKE2SP256 - BLAKE2SP224E - BLAKE2SP224 - SHA1E - SHA1 - MD5E - MD5 - WORM - URL - X* - local repository version: 10 - operating system: darwin x86_64 - remote types: - git - gcrypt - p2p - S3 - bup - directory - rsync - web - bittorrent - webdav - adb - tahoe - glacier - ddar - git-lfs - httpalso - borg - hook - external - supported repository versions: - 8 - 9 - 10 - upgrade supported from repository versions: - 0 - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - version: 10.20230227 ## location - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria - type: dataset ## python - implementation: CPython - version: 3.10.11 ## system - distribution: darwin/20.6.0 10.16/x86_64 - encoding: - default: utf-8 - filesystem: utf-8 - locale.prefered: UTF-8 - filesystem: - CWD: - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria - HOME: - path: /Users/jsheunis - TMP: - path: /var/folders/g7/720mm8ns7hg3d28_3yyzkx_r0000gp/T - max_path_length: 318 - name: Darwin - release: 20.6.0 - type: posix - version: Darwin Kernel Version 20.6.0: Thu Mar 9 20:39:26 PST 2023; root:xnu-7195.141.49.700.6~1/RELEASE_X86_64 ```

datalad / datalad-ria