external special remote protocol error

[mateidavid]

I'm trying to use git-annex-remote-b2 on Ubuntu 14.04 using:

$ git-annex version
git-annex version: 5.20140412ubuntu1
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

I built git-annex-remote-b2 with the go command given in the README, then I copied the executable to /usr/local/bin. Next, I tried to run the test.sh script. I set the B2_* environment variables as required. When I run test.sh, various commands fail with the message external special remote protocol error, unexpectedly received "" (unable to parse command)). First, one of the initremote commands failed this way. I ran it by hand and it worked, then I continued to copy-paste the commands from test.sh. git annex copy failed on the first time, but the second time it ended ok. Then git annex fsck failed. Is there a way to enable more verbose logging to track down what's going on?

[encryptio]

git-annex 5.20140412 is really, really old; notably it doesn't have git annex testremote, so the integration test wouldn't fully work anyway.

That said, I created a docker container with ubuntu 14.04, installed Go 1.6 and git-annex 5.20140412 (both versions which were in apt), and after small fixup to the shebang lines, the integration test worked as well as I could have expected; no errors up until testremote (which fails because it doesn't exist in that git-annex version.)

I just added a GIT_ANNEX_EXTERNAL_B2_PROTOCOL_DEBUG environment variable option; set it to something and it'll print everything it reads from git-annex and everything it writes to git-annex also to stderr (which should go to your terminal.) Try that out and see if it shows anything useful.

[mateidavid]

I first patched test.bash (to decouple building from testing) with: test.bash.patch.txt

I built git-annex-remote-b2 with: GOPATH=/tmp/gopath go get github.com/encryptio/git-annex-remote-b2

I ran the test with: cd /tmp/gopath/src/github.com/encryptio/git-annex-remote-b2; PATH=/tmp/gopath/bin:$PATH GIT_ANNEX_EXTERNAL_B2_PROTOCOL_DEBUG=1 bash -x test.bash |& tee test.log.txt.

Here are 2 different log files I obtained by running the test several times. (I only edited out my B2 creds.) test.log.1.txt test.log.2.txt

As you can see, the failure is in different steps, though I can't really tell what the problem is. I did all this in case you might find it useful, but in light of your comment about git-annex being too old, I moved on to a newer version.

I tried to compile it from source, failed, found this: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/431 (which looked daunting), gave up, and went the Docker route. I installed git-annex=6.20160511-1 in a debian:unstable based container, then golang-go, then git-annex-remote-b2 with GOPATH=/usr/local go get github.com/encryptio/git-annex-remote-b2. I applied the above patch again, moved credentials into the container, ran the tests, and everything went just fine, except for some strange messages saying "Cannot run git-annex-remote-!dne!" for the unavailable remote removeKey and storeKey tests, but those did not crash the test.

[encryptio]

The error message from git-annex implies that the external process is dying and closing its filehandles, and both failures end at the point where git-annex-remote-b2 does its call to B2 to do authentication, which is its first network call. Then, for some reason, the external's stdout gets closed; probably from the process dying, getting killed, or successfully and buggily exiting with no error.

It doesn't look like the implementation of external special remotes has changed notably since 5.20140412, and it doesn't depend on any tricky libraries, so git-annex probably isn't the culprit.

I also don't think it's a race condition inside git-annex-remote-b2, since the code is entirely serial except for the hashing of files (which hasn't run at all by the time the failure happens.) git-annex's special remote implementation is also entirely serial.

After looking over the code, there shouldn't be a way for git-annex-remote-b2 to exit like that without being killed. Maybe some security rules are killing it? OOM killer (run dmesg)? Is it reliable with the new version of git-annex in the same environment (run many times)?

[mateidavid]

I posted details of the Docker walkthrough in here: http://git-annex.branchable.com/install/Docker/#comment-405d13a7864a3b8cdd518490cabcde1d . In addition to what I wrote in there, I also installed git-annex-remote-b2 inside the container (by uncommenting the respective line). I ran the (patched) tests twice with no error whatsoever, and each run involves multiple git annex calls. On the host, I could never get more than 2-3 git annex calls before one of these crashes. So it seems to be reliable under Docker. Nothing noteworthy under /var/log when crashing on the host.