search

spine-generic / data-multi-subject

Multi-subject data for the Spine Generic project
Creative Commons Attribution 4.0 International
21 stars 15 forks source link

enableremote amazon git-annex: Unknown remote type S3 #33

Open jcohenadad opened 3 years ago

jcohenadad commented 3 years ago

Maybe the problem I've been having in https://github.com/spine-generic/data-multi-subject/issues/24 is related to the fact that amazon remote does not seem to be enabled on my repos:

julien-macbook:~/code/spine-generic/data-multi-subject $ git annex info 
trusted repositories: 0
semitrusted repositories: 5
    00000000-0000-0000-0000-000000000001 -- web
    00000000-0000-0000-0000-000000000002 -- bittorrent
    0dc8fc75-8d58-4c09-850d-63ff89d7a5ba -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject
    5a5447a8-a9b8-49bc-8276-01a62632b502 -- amazon
    5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject [here]
untrusted repositories: 0
transfers in progress: none
available local disk space: 14.47 gigabytes (+1 megabyte reserved)
local annex keys: 1983
local annex size: 10.98 gigabytes
annexed files in working tree: 2497
size of annexed files in working tree: 11.02 gigabytes
bloom filter size: 32 mebibytes (0.4% full)
backend usage: 
    SHA256E: 2497

But then:

julien-macbook:~/code/spine-generic/data-multi-subject $ git annex enableremote amazon
enableremote amazon 
git-annex: Unknown remote type S3 (pick from: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external)
failed
git-annex: enableremote: 1 failed

Making sure I have the correct git-annex version:

julien-macbook:~/code/spine-generic/data-multi-subject $ git-annex version
git-annex version: 8.20200810
jcohenadad commented 3 years ago

However, based on this comment https://github.com/spine-generic/data-multi-subject_DO-NOT-USE/issues/20#issuecomment-669456435, with autoenable=true, it seems like git-annex enableremote amazon is not required after all 🤷‍♂️

kousu commented 3 years ago

From https://github.com/spine-generic/data-multi-subject/issues/18#issuecomment-683494397

i'm beginning to think that there might be something wrong with my installed version of git-annex:

git-annex version
git-annex version: 8.20200810
build flags: Assistant Webapp Pairing WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8

This looks indeed like a version problem. You did did brew install git-annex right?

Here's mine, from pacman -S git-annex:

$ git annex version
git-annex version: 8.20200810-g3400b0188
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.2 http-client-0.7.2.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8

Both claim to be the same version, but obviously they're not because they have different plugins compiled in.

This is the build script you would have used: https://github.com/Homebrew/homebrew-core/blob/a80db9e908e36bcf4db0d74a9f3abacc97efa285/Formula/git-annex.rb; here's the one I used: https://github.com/archlinux/svntogit-community/blob/511d422f8aaa8ada1290994dc855c37fb1cacef9/trunk/PKGBUILD. I think the difference is that Arch's says it depends on "haskell-aws".

I'm taking a look on a macOS machine now and I see the same thing you do. This was working for you three weeks ago. Maybe a brew update introduced a regression?

kousu commented 3 years ago

It looks like either https://github.com/Homebrew/homebrew-core/pull/59330/files (switching build scripts) or https://github.com/Homebrew/homebrew-core/pull/59470/files (bumping the version number) caused a regression.

I'm going to try https://stackoverflow.com/a/4158763 to try installing an older version and open a bug report with brew.

In the meantime, can you use conda's version?

jcohenadad commented 3 years ago

Installing older version of brew (thanks @kousu ):

brew uninstall git-annex
cd $(brew --repository)/Library/Taps/homebrew/homebrew-core
git checkout -b git-annex-old 23a4f9fb8513c805fba508f3b769075fb2e090b6^
HOMEBREW_NO_AUTO_UPDATE=1 brew install git-annex

Fixing my local repos:

git-annex copy --to amazon
git-annex sync

Problem: my local copy did not have the actual files (only the symlinks), so i manually copied them from our internal server, and added them to the local repos:

git add -u
# reupload to amazon
git-annex copy --to amazon

Removing dead mirrors:

kousu commented 3 years ago

For the record: no, @jcohenadad can't use conda's version because conda only packages git-annex for linux: https://anaconda.org/conda-forge/git-annex/files (archive, archive).