search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.58k stars 308 forks source link

Git authentication not supported since upgrade to JGit 5.11 #3864

Closed woznik closed 2 years ago

woznik commented 3 years ago

Hello I have been using docker image build from the Dockerfile with the commit 8a44ec6751caaf6e4ad209ae6f7918809733de03 09.03.2021 Yesterday I created a new image from the most recent commit: 9af63f4dde18d63d7bb10fe2e328079ecd9a4c1c 07.04.2021 and suring a scan phase I face a git authentication issue

Caused by: TransportException: https://gitlab-ci-token@*******.git: authentication not supported

I use still the same .netrc file with machine address, gitlab-ci-token and the API_TOKEN. In the debug mode I see that the .netrc file was loaded and there is a proper address of the machine collected

I read the changes and the docs and I didn't find any specific change in that matter.

Was there any specific change to the git authentication mechanisms? Thanks for any help.

tsteenbe commented 3 years ago

@sschuberth Is this maybe due to using Git protocol v2 now, see https://github.com/oss-review-toolkit/ort/pull/3762

sschuberth commented 3 years ago

Actually, that change shouldn't be related as Git still falls back to protocol v1 if v2 is not supported by the server.

woznik commented 3 years ago

Hello Thank for your feedback I have tested that the issue appears on the commit from 18 March on the commit: 9d06698a60b3f583e58c8faa55f623eae40e4690

sschuberth commented 3 years ago

That commit is totally unrelated to anything Git. The cause must then be something else that coincidentally happened at the same time.

woznik commented 3 years ago

Yes, I agree. It does not relate to this specific commit I just indicated the point in time where I find the issue for the first time

sschuberth commented 3 years ago

So, mind bisecting between 8a44ec6751caaf6e4ad209ae6f7918809733de03 and 9d06698a60b3f583e58c8faa55f623eae40e4690 to find out the exact commit that breaks for you?

woznik commented 3 years ago

I found that the authentication not supported appears on my side with this commit: Gradle: Upgrade JGit to version 5.11.0.202103091610-r https://github.com/oss-review-toolkit/ort/commit/df23665ab7c41aaab8643673aee31cbd0e475612#diff-3d103fc7c312a3e136f88e81cef592424b8af2464c468116545c4d22d6edcf19

What I see in the debug log is that the head response from git server side also uses v2, but for some reasons the authentication is not supported

21:49:57.389 [main] DEBUG org.eclipse.jgit.internal.storage.file.FileSnapshot - file=/builds/sqsc/home_cloud/classimage_server/ort/scanner/downloads/Unmanaged/unknown/classimage_server/337c18baddfd981c51ee050b7a37de8ff6fd6c34/.git/config, isRacyClean=true, read=2021-04-13 21:49:57.388839000, lastModified=2021-04-13 21:49:57.000000000, delta=388839000 ns, racy<=2500000000 ns
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< version 2
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< agent=git/2.29.0
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< ls-refs
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< fetch=shallow filter
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< server-option
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< object-format=sha1
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineIn - git< 0000
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> command=ls-refs
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> agent=JGit/5.11.0.202103091610-r
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> 0001
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> peel
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> symrefs
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> ref-prefix refs/tags/
21:49:57.389 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> 0000
21:49:57.468 [main] DEBUG org.eclipse.jgit.transport.PacketLineOut - git> 0000
21:49:57.468 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package 'classimage_server' and version '337c18baddfd981c51ee050b7a37de8ff6fd6c34' found: IOException: Unable to list the remote tags.
Caused by: TransportException: https://***************/sqsc/home_cloud/classimage_server.git: authentication not supported
Caused by: TransportException: https://***************/sqsc/home_cloud/classimage_server.git: authentication not supported
21:49:57.468 [main] DEBUG org.ossreviewtoolkit.downloader.Downloader - VCS download failed for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34': DownloadException: Unable to determine a revision to checkout.
21:49:57.469 [main] PERFORMANCE org.ossreviewtoolkit.downloader.Downloader - Failed attempt to download source code for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34' from VcsInfo(type=Git, url=https://gitlab-ci-token@***************/sqsc/home_cloud/classimage_server.git, revision=337c18baddfd981c51ee050b7a37de8ff6fd6c34, resolvedRevision=null, path=) took 1020.248132ms.
21:49:57.471 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source artifact for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34' from ...
21:49:57.471 [main] DEBUG org.ossreviewtoolkit.downloader.Downloader - Source artifact download failed for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34': DownloadException: No source artifact URL provided for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34'.
21:49:57.471 [main] PERFORMANCE org.ossreviewtoolkit.downloader.Downloader - Failed attempt to download source code for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34' from RemoteArtifact(url=, hash=Hash(value=, algorithm=)) took 0.819767ms.
21:49:57.473 [main] ERROR org.ossreviewtoolkit.scanner.LocalScanner - Could not download 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34': DownloadException: Download failed for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34'.
Suppressed: DownloadException: Unable to determine a revision to checkout.
Suppressed: DownloadException: No source artifact URL provided for 'Unmanaged::classimage_server:337c18baddfd981c51ee050b7a37de8ff6fd6c34'.
$ git --version
git version 2.31.1
$ git config --global protocol.version 2
$ GIT_TRACE_CURL=1 git -c protocol.version=2 ls-remote https://******************/sqsc/home_cloud/classimage_server.git 2>&1 | grep Git-Protocol
22:54:17.359559 http.c:715              => Send header: Git-Protocol: version=2
22:54:17.389185 http.c:715              => Send header: Git-Protocol: version=2
22:54:17.442138 http.c:715              => Send header: Git-Protocol: version=2
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote https://****************/sqsc/home_cloud/classimage_server.git 2>&1 | head
22:54:17.638956 pkt-line.c:80           packet:          git< # service=git-upload-pack
22:54:17.638974 pkt-line.c:80           packet:          git< 0000
22:54:17.638976 pkt-line.c:80           packet:          git< version 2
22:54:17.638980 pkt-line.c:80           packet:          git< agent=git/2.29.0
22:54:17.638982 pkt-line.c:80           packet:          git< ls-refs
22:54:17.638984 pkt-line.c:80           packet:          git< fetch=shallow filter
22:54:17.638985 pkt-line.c:80           packet:          git< server-option
22:54:17.638986 pkt-line.c:80           packet:          git< object-format=sha1
22:54:17.638988 pkt-line.c:80           packet:          git< 0000
22:54:17.639029 pkt-line.c:80           packet:          git< version 2
sschuberth commented 3 years ago

As the JGit 5.11 release notes show several changes to TransportHttp it might be that they broke something... please reach out directly to the JGit community about this, e.g. by asking on their mailing list, or filing a bug on their side.

sschuberth commented 3 years ago

Also, do you happen to know which authentication scheme the server is expecting? Is it Kerberos?

woznik commented 3 years ago

I tested some config changes to force v1 to be used by the client but no use finally I did a downgrade of JGit to 5.10 and it works on my side. I will check the authentication scheme with the admins. I will raise this issue to JGit community

woznik commented 3 years ago

Also, do you happen to know which authentication scheme the server is expecting? Is it Kerberos?

Here is what I get from the admins: For authentication (users mainly use graphical user interface) we use SAML auth solution with Lemonldap. For Git auth there are 2 possibles in https with username and a personnal access token or ssh key.

sschuberth commented 3 years ago

There is a chance that this got fixed by https://github.com/oss-review-toolkit/ort/pull/4315. Could you please verify, @woznik?

sschuberth commented 3 years ago

I will raise this issue to JGit community

If you ever did this, mind sharing a link?

sschuberth commented 2 years ago

Feel free to re-open if this issue still occurs.