Unable to push branches to mcap repo since addition of .lfsconfig

[wkalt]

Since commit 92d814fdaa372e2b7d268c813a6428b07043d22b, which added the .lfsconfig file, I have been unable to push branches to the mcap repo. I successfully did it one time on a fresh clone, but now that isn't working either, so I suspect that previous test might have been a mixup.

Some info:

[~/work/mcap3] (task/test-branch) $ git --version
git version 2.36.1
[~/work/mcap3] (task/test-branch) $ git-lfs --version
git-lfs/3.0.1 (GitHub; linux amd64; go 1.17.1)

[~/work/mcap3] (task/test-branch) $ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = git@github.com:foxglove/mcap
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[lfs]
    repositoryformatversion = 0
[lfs "https://github.com/foxglove/mcap.git/info/lfs"]
    access = basic

[~/work/mcap3] (task/test-branch) $ cat ~/.gitconfig
# This is Git's per-user configuration file.
[user]
# Please adapt and uncomment the following lines:
name = Wyatt Alt
email = wyatt.alt@gmail.com
[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true
[gist]
    private = true

Here's the debug output when I try and push a branch from a fresh clone, with a single empty commit:

[~/work/mcap3] (task/test-branch) $ GIT_TRACE=true git push origin task/test-branch
20:25:16.191667 git.c:459               trace: built-in: git push origin task/test-branch
20:25:16.192427 run-command.c:654       trace: run_command: unset GIT_PREFIX; ssh git@github.com 'git-receive-pack '\''foxglove/mcap'\'''
20:25:18.260378 run-command.c:654       trace: run_command: .git/hooks/pre-push origin git@github.com:foxglove/mcap
20:25:18.263646 git.c:748               trace: exec: git-lfs pre-push origin git@github.com:foxglove/mcap
20:25:18.263705 run-command.c:654       trace: run_command: git-lfs pre-push origin git@github.com:foxglove/mcap
20:25:18.266362 trace git-lfs: exec: git 'version'
20:25:18.267483 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
20:25:18.268177 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:25:18.268803 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:25:18.269526 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:25:18.270149 trace git-lfs: exec: git 'config' '--includes' '-l'
20:25:18.270824 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:25:18.271559 trace git-lfs: exec: git 'config' '--includes' '-l' '-f' '/home/wyatt/work/mcap3/.lfsconfig'
20:25:18.272463 trace git-lfs: pre-push: refs/heads/task/test-branch 1be328d0343eb4c2b3d77c15d09287c32667ecc9 refs/heads/task/test-branch 0000000000000000000000000000000000000000
20:25:18.272498 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:25:18.273697 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
20:25:21.614759 trace git-lfs: creds: git credential fill ("https", "github.com", "")
20:25:21.614818 trace git-lfs: exec: git 'credential' 'fill'
Username for 'https://github.com':

The issue seems to be LFS using https instead of ssh authentication. Note the push itself is using ssh and my ssh works fine:

[~/work/mcap3] (task/test-branch) $ ssh git@github.com
PTY allocation request failed on channel 0
Hi wkalt! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

If I remove the .lfsconfig file, push works again.

[james-rms]

Here's what I see:

j@192-168-1-108 mcap % git remote get-url origin
git@github.com:foxglove/mcap
j@192-168-1-108 mcap % cat ~/.gitconfig
[user]
    name = James Smith
    email = james@foxglove.dev
[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true
[gist]
    private = true
j@192-168-1-108 mcap % cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = git@github.com:foxglove/mcap
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
    remote = origin
    merge = refs/heads/main
[branch "jrms/bugfix/ros1msg-field-parse-whitespace"]
    remote = origin
    merge = refs/heads/jrms/bugfix/ros1msg-field-parse-whitespace
[lfs]
    repositoryformatversion = 0
j@192-168-1-108 mcap % GIT_TRACE=true git push origin brungs
15:34:25.106377 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/bin/git
15:34:25.106701 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/bin
15:34:25.107066 git.c:455               trace: built-in: git push origin brungs
15:34:25.118655 run-command.c:667       trace: run_command: unset GIT_PREFIX; ssh git@github.com 'git-receive-pack '\''foxglove/mcap'\'''
15:34:27.612197 run-command.c:667       trace: run_command: .git/hooks/pre-push origin git@github.com:foxglove/mcap
15:34:27.623669 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
15:34:27.624013 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
15:34:27.624327 git.c:743               trace: exec: git-lfs pre-push origin git@github.com:foxglove/mcap
15:34:27.624338 run-command.c:667       trace: run_command: git-lfs pre-push origin git@github.com:foxglove/mcap
15:34:27.629122 trace git-lfs: exec: git 'version'
15:34:27.632261 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
15:34:27.635332 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
15:34:27.637950 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
15:34:27.641101 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
15:34:27.644799 trace git-lfs: exec: git 'config' '--includes' '-l'
15:34:27.647835 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
15:34:27.650965 trace git-lfs: exec: git 'config' '--includes' '-l' '-f' '/Users/j/code/mcap/.lfsconfig'
15:34:27.654718 trace git-lfs: pre-push: refs/heads/brungs 283972f1055442bc1263b8dcf5d1f06fcce908a1 refs/heads/brungs 0000000000000000000000000000000000000000
15:34:27.654926 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
15:34:27.660563 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
15:34:30.366763 trace git-lfs: creds: git credential fill ("https", "github.com", "")
15:34:30.366915 trace git-lfs: exec: git 'credential' 'fill'
15:34:30.404245 trace git-lfs: Filled credentials for https://github.com/foxglove/mcap.git/info/lfs
15:34:30.405675 trace git-lfs: exec: /usr/bin/security 'list-keychains'
15:34:30.420528 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
15:34:30.433793 trace git-lfs: HTTP: POST https://github.com/foxglove/mcap.git/info/lfs/locks/verify
15:34:30.976440 trace git-lfs: HTTP: 200
15:34:30.976540 trace git-lfs: creds: git credential approve ("https", "github.com", "")
15:34:30.976815 trace git-lfs: exec: git 'credential' 'approve'
15:34:31.037961 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

15:34:31.038540 trace git-lfs: tq: running as batched queue, batch size of 100
15:34:31.038713 trace git-lfs: run_command: git rev-list --objects --ignore-missing --stdin --
15:34:31.038722 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-list' '--objects' '--ignore-missing' '--stdin' '--'
15:34:31.039916 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
15:34:31.041038 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
15:34:31.046954 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
15:34:31.046972 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
15:34:31.047007 trace git-lfs: filepathfilter: accepting "tmp"
15:34:31.048268 run-command.c:667       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
15:34:31.050438 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
15:34:31.050745 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
15:34:31.051086 git.c:455               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a pull request for 'brungs' on GitHub by visiting:
remote:      https://github.com/foxglove/mcap/pull/new/brungs
remote:
To github.com:foxglove/mcap
 * [new branch]          brungs -> brungs
j@192-168-1-108 mcap %

[james-rms]

Maybe I have something in my keychain which allows this to work?

[wkalt]

Maybe. I notice I have this line in my .git/config file:

[lfs "https://github.com/foxglove/mcap.git/info/lfs"]
    access = basic

You don't have that one. When I remove that line, running git push adds it back in and fails again the same way.

[james-rms]

Aah, i accidentally edited that out. (there's actually a lot more in my .git/config, but it's just branch remote rules.

[lfs "https://github.com/foxglove/mcap.git/info/lfs"]
        access = basic

[james-rms]

Curious if using the github credential manager solves this for you: https://docs.github.com/en/get-started/getting-started-with-git/caching-your-github-credentials-in-git

[james-rms]

@wkalt what does your GIT_TRACE look like with .lfsconfig removed?

[wkalt]

For the GCM question - no dice afaict. I installed the gh command line tool, successfully authenticated with gh auth login, then installed their GCM .deb package from https://github.com/GitCredentialManager/git-credential-manager/releases/tag/v2.0.877 and set export GCM_CREDENTIAL_STORE=secretservice according to the instructions in https://github.com/GitCredentialManager/git-credential-manager/blob/release/docs/credstores.md. Pushing after doing that failed the same way. Unclear if there are additional steps required.

With .lfsconfig gone, here is the trace:

[~/work/mcap3] (task/test-branch) $ rm .lfsconfig
[~/work/mcap3] (task/test-branch) $ git commit -am "remove lfsconfig"
[task/test-branch f4195754e] remove lfsconfig
 1 file changed, 6 deletions(-)
 delete mode 100644 .lfsconfig
[~/work/mcap3] (task/test-branch) $ GIT_TRACE=true git push origin task/test-branch
21:03:23.809622 git.c:459               trace: built-in: git push origin task/test-branch
21:03:23.810435 run-command.c:654       trace: run_command: unset GIT_PREFIX; ssh git@github.com 'git-receive-pack '\''foxglove/mcap'\'''
21:03:25.901790 run-command.c:654       trace: run_command: .git/hooks/pre-push origin git@github.com:foxglove/mcap
21:03:25.904262 git.c:748               trace: exec: git-lfs pre-push origin git@github.com:foxglove/mcap
21:03:25.904323 run-command.c:654       trace: run_command: git-lfs pre-push origin git@github.com:foxglove/mcap
21:03:25.907799 trace git-lfs: exec: git 'version'
21:03:25.909498 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
21:03:25.910969 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
21:03:25.912435 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
21:03:25.913999 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
21:03:25.915367 trace git-lfs: exec: git 'config' '--includes' '-l'
21:03:25.916732 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
21:03:25.917980 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
21:03:25.919467 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
21:03:25.921010 trace git-lfs: attempting pure SSH protocol connection
21:03:25.921042 trace git-lfs: run_command: ssh -- git@github.com git-lfs-transfer foxglove/mcap upload
21:03:25.921154 trace git-lfs: exec: ssh '--' 'git@github.com' 'git-lfs-transfer foxglove/mcap upload'
21:03:28.049426 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF
21:03:28.049863 trace git-lfs: pre-push: refs/heads/task/test-branch f4195754eea595ffac2f4c0b918044a54d4f5b7d refs/heads/task/test-branch 0000000000000000000000000000000000000000
21:03:28.050012 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
21:03:28.052773 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
21:03:30.815636 trace git-lfs: attempting pure SSH protocol connection
21:03:30.815690 trace git-lfs: run_command: ssh -- git@github.com git-lfs-transfer foxglove/mcap upload
21:03:30.815912 trace git-lfs: exec: ssh '--' 'git@github.com' 'git-lfs-transfer foxglove/mcap upload'
21:03:32.964428 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF
21:03:32.964565 trace git-lfs: run_command: ssh -- git@github.com git-lfs-authenticate foxglove/mcap upload
21:03:32.964685 trace git-lfs: exec: ssh '--' 'git@github.com' 'git-lfs-authenticate foxglove/mcap upload'
21:03:35.423730 trace git-lfs: HTTP: POST https://lfs.github.com/foxglove/mcap/locks/verify
21:03:36.343892 trace git-lfs: HTTP: 200
21:03:36.343979 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

21:03:36.351261 trace git-lfs: tq: running as batched queue, batch size of 100
21:03:36.351458 trace git-lfs: run_command: git rev-list --objects --ignore-missing --stdin --
21:03:36.352057 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
21:03:36.352508 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
21:03:36.357895 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
21:03:36.357931 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
21:03:36.357970 trace git-lfs: filepathfilter: accepting "tmp"
21:03:36.359901 run-command.c:654       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
21:03:36.361029 git.c:459               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 381 bytes | 381.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote:
remote: Create a pull request for 'task/test-branch' on GitHub by visiting:
remote:      https://github.com/foxglove/mcap/pull/new/task/test-branch
remote:
To github.com:foxglove/mcap
 * [new branch]          task/test-branch -> task/test-branch

[wkalt]

If I change the .lfsconfig file to read

[lfs]
    # Specify the server to use when pulling Git LFS assets.
    # This is here to work around a bug in the Swift Package Manager which prevents
    # it from installing packages when the source repository uses Git LFS.
    # https://forums.swift.org/t/swiftpm-with-git-lfs/42396/4
    url = git@github.com:foxglove/mcap.git

everything works as I would expect. I don't know if that invalidates the swift fix though.

[james-rms]

I think this would also make life difficult for https users.