kousu commented 3 years ago

Bug report

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?
[x] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link? Error: no logs ?
[x] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

brew config

``` user@users-iMac homebrew-core % brew config HOMEBREW_VERSION: 2.4.16 ORIGIN: https://github.com/Homebrew/brew HEAD: 2a4c9f610e69b9fed1eec61ac2c53e4b11a32e4c Last commit: 29 hours ago Core tap ORIGIN: https://github.com/Homebrew/homebrew-core Core tap HEAD: 41a4db231ecd421c896b57ad802611c590ab33a2 Core tap last commit: 15 minutes ago Core tap branch: master HOMEBREW_PREFIX: /usr/local HOMEBREW_CASK_OPTS: [] HOMEBREW_MAKE_JOBS: 4 CPU: quad-core 64-bit penryn Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby Clang: 11.0 build 1103 Git: 2.24.2 => /Library/Developer/CommandLineTools/usr/bin/git Curl: 7.64.1 => /usr/bin/curl macOS: 10.15.4-x86_64 CLT: 11.4.1.0.1.1586360307 Xcode: N/A user@users-iMac homebrew-core % brew doctor Your system is ready to brew. ```

What you were trying to do (and why)

Use git-annex to share scientific data.

What happened (include command output)

We have a dataset whose content is on AWS, via git-annex (https://github.com/spine-generic/data-multi-subject/); all the datasets in https://openneuro.org and most in https://portal.conp.ca are the same.

But since the middle of the month or so this has been happening:

Command output

``` julien-macbook:~/code/spine-generic/data-multi-subject $ git checkout -b jca/17-defacing Switched to a new branch 'jca/17-defacing' julien-macbook:~/code/spine-generic/data-multi-subject $ git remote amazon origin julien-macbook:~/code/spine-generic/data-multi-subject $ julien-macbook:~/code/spine-generic/data-multi-subject $ gs M sub-cmrra02/anat/sub-cmrra02_T2w.nii.gz M sub-cmrra04/anat/sub-cmrra04_T2w.nii.gz M sub-cmrra05/anat/sub-cmrra05_T2w.nii.gz M sub-oxfordFmrib07/anat/sub-oxfordFmrib07_T1w.nii.gz M sub-stanford06/anat/sub-stanford06_T1w.nii.gz julien-macbook:~/code/spine-generic/data-multi-subject $ gc (recording state in git...) [jca/17-defacing 4ea269a7] Fixed defacing in a few subjects 5 files changed, 5 insertions(+), 5 deletions(-) julien-macbook:~/code/spine-generic/data-multi-subject $ git annex sync --content amazon git-annex: there is no available git remote named "amazon" julien-macbook:~/code/spine-generic/data-multi-subject $ git remote amazon origin julien-macbook:~/code/spine-generic/data-multi-subject $ ``` and this: ``` julien-macbook:~/code/spine-generic/data-multi-subject $ git annex enableremote amazon enableremote amazon git-annex: Unknown remote type S3 (pick from: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external) failed git-annex: enableremote: 1 failed ``` The most underlying issue is that git-annex has lost its S3 support. The telling symptom is that this doesn't say "S3" or "aws" anywhere: ``` git annex version git-annex version: 8.20200720.1 build flags: Assistant Webapp Pairing WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite dependency versions: bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2 key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external operating system: darwin x86_64 supported repository versions: 8 upgrade supported from repository versions: 0 1 2 3 4 5 6 7 ```

What you expected to happen

A working copy should say "build flags: ... S3 ..." and "dependency versions: aws-0.22":

git annex version
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

This was working for us a month ago.

Step-by-step reproduction instructions (by running `brew install` commands)

brew update
brew install git-annex
git annex version

claui commented 3 years ago

Pinging fellow maintainer @chenrui333, who has authored #59330 and may be able to help.

kousu commented 3 years ago

Oh great, thanks for checking in so quick. I do indeed think it was #59330

log for git-annex.rb

user@users-iMac ~ % cd $(brew --repository)/Library/Taps/homebrew/homebrew-core user@users-iMac homebrew-core % git log -p master -- Formula/git-annex.rb commit 9591758fd8a565abf98d13ffd87c66e5e12ad579 Author: Nanda H Krishna Date: Fri Aug 28 21:06:02 2020 +0530 Add livecheck blocks from homebrew-livecheck diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb index 1de6a6ccd6..8aa02aaeea 100644 --- a/Formula/git-annex.rb +++ b/Formula/git-annex.rb @@ -5,6 +5,10 @@ class GitAnnex < Formula sha256 "e631c9d52e440f80e9d305c95a078dcae71f200125bca91e49d5b8e2d864c6f3" head "git://git-annex.branchable.com/" + livecheck do + url :stable + end + bottle do cellar :any sha256 "b2da7a293056e721b82cd6cfbc706203611321677ae78f084d24531df3423b04" => :catalina commit 7a99e3cb8d541d62cdb9a423b6141db7b90cd682 Author: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com> Date: Wed Aug 12 01:13:12 2020 +0000 git-annex: update 8.20200810 bottle. diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb index b4dfc9b309..1de6a6ccd6 100644 --- a/Formula/git-annex.rb +++ b/Formula/git-annex.rb @@ -7,9 +7,9 @@ class GitAnnex < Formula bottle do cellar :any - sha256 "f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5" => :catalina - sha256 "2fa4166aa125e2c0e620b18506ddd7649d4ae4baf8bc22bb9b29d167ca33a04a" => :mojave - sha256 "33d83e7d1dca6016fea3539c5f0daa6bbba831f7eaa3f69301d68f8da7b060d4" => :high_sierra + sha256 "b2da7a293056e721b82cd6cfbc706203611321677ae78f084d24531df3423b04" => :catalina + sha256 "cb3c028e6f987813c02dd7d1e3382721896ddfce92924776d3715e1912323766" => :mojave + sha256 "969571512b9bbf02b5e1492a0b36501b8d3050e1c243c322accfc18eeba4dbe2" => :high_sierra end depends_on "cabal-install" => :build commit 7e90adf034c134b895fee3983f5ba2d017628843 Author: Nick Golder Date: Tue Aug 11 06:55:22 2020 -0600 git-annex 8.20200810 Closes #59470. Signed-off-by: chenrui Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com> diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb index 7ee16b17e2..b4dfc9b309 100644 --- a/Formula/git-annex.rb +++ b/Formula/git-annex.rb @@ -1,9 +1,8 @@ class GitAnnex < Formula desc "Manage files with git without checking in file contents" homepage "https://git-annex.branchable.com/" - url "https://hackage.haskell.org/package/git-annex-8.20200720.1/git-annex-8.20200720.1.tar.gz" - sha256 "d6252697151ab01cc8f7a492dc5eb4b4e773c0d48771df360c82a7cca7a39c3c" - revision 1 + url "https://hackage.haskell.org/package/git-annex-8.20200810/git-annex-8.20200810.tar.gz" + sha256 "e631c9d52e440f80e9d305c95a078dcae71f200125bca91e49d5b8e2d864c6f3" head "git://git-annex.branchable.com/" bottle do commit 6370ed052ee3ca595f77bbf5988689ddec04ff17 Author: chenrui Date: Sat Aug 8 17:39:58 2020 +0000 git-annex: update 8.20200720.1_1 bottle. diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb index f3f5ee35e6..7ee16b17e2 100644 --- a/Formula/git-annex.rb +++ b/Formula/git-annex.rb @@ -8,9 +8,9 @@ class GitAnnex < Formula bottle do cellar :any - sha256 "0dc6cab32110246d38e26753141ac0772557cf303cf7a92f52edf1b4b2c9f625" => :catalina - sha256 "15cc50337736d3a405ca28665dc89e850e29d622a7233d0071249fa5fbe66eba" => :mojave - sha256 "215ae839c10f9fa39a236d0f7cd3bd9630b6c719a2d9e593bdd03b692c6bbc6e" => :high_sierra + sha256 "f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5" => :catalina + sha256 "2fa4166aa125e2c0e620b18506ddd7649d4ae4baf8bc22bb9b29d167ca33a04a" => :mojave + sha256 "33d83e7d1dca6016fea3539c5f0daa6bbba831f7eaa3f69301d68f8da7b060d4" => :high_sierra end depends_on "cabal-install" => :build commit 23a4f9fb8513c805fba508f3b769075fb2e090b6 Author: Rui Chen Date: Sat Aug 8 09:19:11 2020 -0400 git-annex: deprecate cabal v1 Closes #59330. Signed-off-by: chenrui diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb index 0523bb2604..f3f5ee35e6 100644 --- a/Formula/git-annex.rb +++ b/Formula/git-annex.rb @@ -1,12 +1,9 @@ -require "language/haskell" - class GitAnnex < Formula - include Language::Haskell::Cabal - desc "Manage files with git without checking in file contents" homepage "https://git-annex.branchable.com/" url "https://hackage.haskell.org/package/git-annex-8.20200720.1/git-annex-8.20200720.1.tar.gz" sha256 "d6252697151ab01cc8f7a492dc5eb4b4e773c0d48771df360c82a7cca7a39c3c" + revision 1 head "git://git-annex.branchable.com/" bottle do @@ -24,10 +21,8 @@ class GitAnnex < Formula depends_on "quvi" def install - install_cabal_package "--constraint", "http-conduit>=2.3", - "--constraint", "network>=2.6.3.0", - using: ["alex", "happy", "c2hs"], - flags: ["s3", "webapp"] + system "cabal", "v2-update" + system "cabal", "v2-install", *std_cabal_v2_args bin.install_symlink "git-annex" => "git-annex-shell" end

The version before PR #59330, 8.20200720.1, is working:

user@users-iMac homebrew-core % git checkout -b git-annex-old 23a4f9fb8513c805fba508f3b769075fb2e090b6^ # note this is the step *BEFORE* the PR #59330
Updating files: 100% (2606/2606), done.
Switched to a new branch 'git-annex-old'
user@users-iMac homebrew-core % HOMEBREW_NO_AUTO_UPDATE=1 brew install git-annex                       
==> Downloading https://homebrew.bintray.com/bottles/git-annex-8.20200720.1.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/0dc6cab32110246d38e26753141ac0772557cf303cf7a92f52edf1b4b2c9f625?response-content-disposition=attachment%3Bfilename%3D%2
######################################################################## 100.0%
==> Pouring git-annex-8.20200720.1.catalina.bottle.tar.gz
==> Caveats
To have launchd start git-annex now and restart at login:
  brew services start git-annex
Or, if you don't want/need a background service you can just run:
  git annex assistant --autostart
==> Summary
🍺  /usr/local/Cellar/git-annex/8.20200720.1: 113 files, 42.5MB
user@users-iMac homebrew-core % git annex version
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

while the version before the next PR #59470, git-annex-8.20200720.1_1, is not:

user@users-iMac homebrew-core % git reset --hard 7e90adf034c134b895fee3983f5ba2d017628843^
Updating files: 100% (2576/2576), done.
HEAD is now at 65be49a3c0 helm: update 3.3.0 bottle.
user@users-iMac homebrew-core % brew uninstall git-annex                                  
Uninstalling /usr/local/Cellar/git-annex/8.20200810... (9 files, 53.3MB)
user@users-iMac homebrew-core % HOMEBREW_NO_AUTO_UPDATE=1 brew install git-annex          
==> Downloading https://homebrew.bintray.com/bottles/git-annex-8.20200720.1_1.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5?response-content-disposition=attachment%3Bfilename%3D%2
######################################################################## 100.0%
==> Pouring git-annex-8.20200720.1_1.catalina.bottle.tar.gz
==> Caveats
To have launchd start git-annex now and restart at login:
  brew services start git-annex
Or, if you don't want/need a background service you can just run:
  git annex assistant --autostart
==> Summary
🍺  /usr/local/Cellar/git-annex/8.20200720.1_1: 9 files, 53.2MB
user@users-iMac homebrew-core % git annex version                                         
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

I don't know about cabal but I can try to fix it.

For comparison, Arch uses the included Makefile instead of cabal, and makes sure to depend on haskell-aws:

https://github.com/archlinux/svntogit-community/blob/511d422f8aaa8ada1290994dc855c37fb1cacef9/trunk/PKGBUILD#L36-L41

kousu commented 3 years ago

This caused https://github.com/spine-generic/data-multi-subject/issues/24 and https://github.com/spine-generic/data-multi-subject/issues/33 and https://github.com/spine-generic/data-multi-subject/issues/18. They're all actually the same, it's just that git-annex isn't that clear about reporting errors.