Pushing to cern gitlab with kerberos authentification does not work with git from Key4hep

[Zehvogel]

with the system git it does. Maybe broken kerberos lib in stack or missing rpath somewhere?

Some more info:

[lreichen@pcphsft121 OpenDataDetector]$ git push -u origin fix-plugins
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.cern.ch:8443/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.cern.ch:8443/lreichen/OpenDataDetector.git/'

[lreichen@pcphsft121 OpenDataDetector]$ git remote -v
origin  https://:@gitlab.cern.ch:8443/lreichen/OpenDataDetector.git (fetch)
origin  https://:@gitlab.cern.ch:8443/lreichen/OpenDataDetector.git (push)
upstream        https://gitlab.cern.ch/acts/OpenDataDetector.git (fetch)
upstream        https://gitlab.cern.ch/acts/OpenDataDetector.git (push)

[lreichen@pcphsft121 OpenDataDetector]$ which git
/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-10-06-paestum/x86_64-almalinux9-gcc11.3.1-opt/git/2.41.0-2s6veu/bin/git

[lreichen@pcphsft121 OpenDataDetector]$ echo $GIT_EXEC_PATH

[lreichen@pcphsft121 OpenDataDetector]$ GIT_TRACE=1 git push
15:06:21.344406 git.c:462               trace: built-in: git push
15:06:21.344573 run-command.c:661       trace: run_command: GIT_DIR=.git git remote-https origin https://:@gitlab.cern.ch:8443/lreichen/OpenDataDetector.git
15:06:21.346593 git.c:748               trace: exec: git-remote-https origin https://:@gitlab.cern.ch:8443/lreichen/OpenDataDetector.git
15:06:21.346630 run-command.c:661       trace: run_command: git-remote-https origin https://:@gitlab.cern.ch:8443/lreichen/OpenDataDetector.git
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.cern.ch:8443/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.cern.ch:8443/lreichen/OpenDataDetector.git/'

Things tried so far without success:

git config --global http.emptyAuth true

export GIT_EXEC_PATH=/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-10-06-paestum/x86_64-almalinux9-gcc11.3.1-opt/git/2.41.0-2s6veu/libexec/git-core/

[jmcarcell]

Kerberos authentication wasn't working after sourcing the key4hep releases or nightlies but this has been fixed by using kerberos from the system (assuming that in every place you would like to use it has been installed). This can be seen by doing klist after sourcing the release or nightlies that ship their own version of kerberos and seeing how the ticket "disappears" after sourcing. Give it a try to now.

[jmcarcell]

Closing as completed until someone complains

[Zehvogel]

It still does not work, same error message

[jmcarcell]

With the nightlies?

[Zehvogel]

yes

[jmcarcell]

The workaround for this is to do /usr/bin/git pull or whatever you need to do when interacting with the remote. At least that works for me in lxplus on Alma9, I'm not sure it will work everywhere since it may depend on the version of git installed in the system.

[jmcarcell]

Can you tell me if this is still an issue @Zehvogel? Or how to reproduce? I don't seem to be able to push to my forks with kerberos with the system git nor with the key4hep git

EDIT: It seems this is caused by the configuration of the ssh provided by the key4hep stack not being the same as the host system; I'll have a look at fixing this and if possible we won't provide ssh in the stack to avoid this

[jmcarcell]

This is now fixed, the answer was both to use the system ssh for ssh authentication and for http authentication to build curl (the system one can't be used because something else wants a newer version) with Kerberos support: https://github.com/key4hep/key4hep-spack/pull/567/files#diff-2e8985f6551ac23a3a88f1dae971052a0160e00b3c0d9048b836df9744911d31R7-R8 Now

$ curl --version
curl 8.6.0 (x86_64-pc-linux-gnu) libcurl/8.6.0 OpenSSL/3.2.1 zlib/1.3.0.zlib-ng libidn2/2.3.4 nghttp2/1.57.0
Release-Date: 2024-01-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets

And GSS-API and Kerberos is in Features