Git pull stuck on Initializing Keybase... done. under Windows with cygwin

[fjouault]

After pushing a repository from Linux, I cannot pull it from Windows with cygwin because git gets stuck after:

$ git pull Initializing Keybase... done.

[strib]

@fjouault: Can you please do a keybase log send and post the resulting log ID here? I don't think we've done any testing on cygwin. Is it possible you didn't start the background keybase service?

[fjouault]

@strib Here is the resulting log ID : ca4c94b10c8c8916be0b851c. Note that I cleared (or at least attempted to clear) all logs before trying again, then sending you the logs. You may therefore only see a couple of attemps.

However, I tried git pull multiple times, including after closing and reopening Windows sessions. Each time the Keybase app and KBFS seem to work correctly, so I assumed every service was launched. Is there something I can try to check if a service needs to be started?

By the way, I got stuck with keybase log send from the cygwin bash prompt: it seemed stuck after the message "This command will send recent keybase log entries to keybase.io for debugging purposes only.

These logs don’t include your private keys or encrypted data, but they will include filenames and other metadata keybase normally can’t read, for debugging purposes.", not showing anything else. I understood what was supposed to happen when running keybase log send from a cmd.exe prompt: "Continue sending logs to keybase.io? (type 'YES' to confirm):" did show then, but not with cygwin. So, I suppose that is at least one issue related to cygwin.

[strib]

@fjouault: I took a look at the logs. It's very strange, it seems like your local git process sends us a "capabilities" command, which is the first step in the protocol. I presume we respond to it with our list of capabilities, and then, the log just stops. Usually there is another step where the local git process sends a few options or starts the pull/push process.

I'm not sure what could be wrong, it seems like git just stopped participating in the protocol. Does git work for other endpoints in that cygwin setup? The only relevant link I could find is this: https://superuser.com/questions/411789/git-clone-does-not-work-in-using-cygwin-to-run-git

(By the way, I notice you're doing a "pull" and not a "clone" -- did "clone" work previously?)

You could also try enabling some of the GIT_TRACE environment variables, and see if anything interesting shows up in the logs: https://git-scm.com/book/gr/v2/Git-Internals-Environment-Variables

[fjouault]

@strib The reason I started with git pull rather than git clone is that I started from an existing repository in which I changed the remote from an ssh-based one to a keybase-based one.

In order to investigate further, I tried cloning, which failed because GIT_DIR is then a cygwin path, not a windows path. Therefore, I tried to wrap git-remote-keybase with a shell script that translates GIT_DIR into a valid Windows path using cygpath -w. This progresses until it reaches the same step it reached with git pull earlier.

If I use git-remote-keybase from the cygwin bash, in interactive mode, I can type commands and get answers (even if i wait some time between commands). However, when git-remote-keybase is run from my wrapper script or directly from git, it does not seem to get the next command. Actually, if I do: $ ( echo capabilities ; echo option verbosity 1 )| GIT_DIR="..." git-remote-keybase origin keybase://...

I get:

fetch
push
option

ok

as expected. But if I add a significant delay between the two echo commands, then git-remote-keybase gets stuck: $ ( echo capabilities ; sleep 3 ; echo option verbosity 1 )| GIT_DIR="..." git-remote-keybase origin keybase://...

yields:

fetch
push
option

Looking at the logs, in the first case (without the sleep) git-remote-keybase did receive the option command, but did not receive it in the second case (with the sleep). It still works with sleep 1 but not with sleep 3. 3 seconds seems to be slightly more than the time keybase takes to initialize. So, it seems that git-remote-keybase gets stuck if it reads its input when no command is there yet. If a commands arrives later, it is too late.

Because this only happens when git-remote-keybase is launched from another cygwin command (git, or a shell script), I suppose this has to do with how cygwin handles pipes.

Not sure what to try next.

[fjouault]

TL;DR: cygwin git is basically not compatible with Keybase, only a "dirty" workaround has been found so far

After further investigation, it seems that calling native Windows executables from cygwin is not supported. It works in some (simple) cases, but generally fails when trying to pipe something in (e.g., when git tries to write commands to stdin of git-remote-keybase). Here is a rough sketch of what happens when git uses git-remote-keybase:

git  --std{in,out}--  git-remote-keybase

The issue is that git is a cygwin executable with cygwin stdin&stdout, whereas git-remote-keybase is a Windows executable with Windows stdin&stdout.

Trying to recompile git-remote-keybase as a cygwin executable would hit the problem that go does not really support cygwin.

I currently believe that the best (although not ideal) way to "resolve" this issue is to explicitly state that cygwin git is not supported by Keybase git.

The only workaround I managed to actually get to work consists in fully separating the cygwin and Windows worlds by a TCP socket. Here is a rough sketch of how everything is connected:

git  --cygwin-std{in,out}--  nc  --TCP--  ncat  --win-std{in,out}--  git-remote-keybase

where git & nc are cygwin executables connected by cygwin std{in,out}, and ncat (from nmap Windows build) & git-remote-keybase are Windows executables connected by Windows std{in,out}. Therefore, std{in,out} connections are only made within each worlds, not between them. The connection between nc and ncat works because it is a simple TCP socket. Here is a sample script (called git-remote-keybase2) that connects everything as described above:

#!/bin/bash

ncat.exe -l -p 1234 --sh-exec "set GIT_DIR=`cygpath -w "$GIT_DIR"`&& git-remote-keybase \"$1\" \"keybase${2#keybase2}\"" &
nc localhost 1234

Note that this script converts the GIT_DIR environment variable from a cygwin path to a Windows path. It also converts the remote from keybase2://... (used to make sure git invokes this script, which is called git-remote-keybase2) to keybase://... (as required by git-remote-keybase).

Remark: making this script work is not necessarily simple. For instance, it assumes that all commands are available and in the PATH. It is also not robust at all (e.g., it uses a hardcoded TCP port). See it as nothing more than a proof-of-concept. I would not recommend actually using it.

[strib]

Thanks for the investigation! cc: @akalin-keybase, who was looking into something similar in the Windows Unix subsystem, I think.

[taruti]

Windows+Cygwin+pipes is prone to issues :(