search

oss-review-toolkit / ort

A suite of tools to automate software compliance checks.
https://oss-review-toolkit.org
Apache License 2.0
1.6k stars 309 forks source link

[Docker] git clone via SSH fails #5894

Closed fb33 closed 2 years ago

fb33 commented 2 years ago

Hi,

Since I updated the source code from August 19 to September 29 for my docker ort image, I've some troubles during the scan.

Now, when I'm running ORT, I add -u ort:ort, but I'm facing with some errors during git clone due to the user.

[2022-09-30T15:32:59.676Z] ________ _____________________
[2022-09-30T15:32:59.676Z] \_____  \\______   \__    ___/ the OSS Review Toolkit, version DOCKER-SNAPSHOT.
[2022-09-30T15:32:59.676Z]  /   |   \|       _/ |    |
[2022-09-30T15:32:59.676Z] /    |    \    |   \ |    |    Running 'scan' under Java 11.0.16.1 on Linux with
[2022-09-30T15:32:59.676Z] \_______  /____|_  / |____|    3 CPUs and a maximum of 10240 MiB of memory.
[2022-09-30T15:32:59.676Z]         \/       \/
[2022-09-30T15:32:59.676Z] Environment variables:
[2022-09-30T15:32:59.676Z] ORT_CONFIG_DIR = /home/ort/.ort/config
[2022-09-30T15:32:59.676Z] ORT_DATA_DIR = /home/ort/.ort
[2022-09-30T15:32:59.676Z] JAVA_HOME = /opt/java/openjdk
[2022-09-30T15:32:59.676Z] ANDROID_HOME = /opt/android-sdk
[2022-09-30T15:32:59.676Z] 
[...]
[2022-09-30T15:33:34.121Z] 15:33:31.596 [main] INFO  org.ossreviewtoolkit.scanner.PathScanner - Scanning 'Maven:com.fasterxml:classmate:1.5.1' in thread 'main' (2 of 79)
[2022-09-30T15:33:34.121Z] 15:33:31.599 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'Maven:com.fasterxml:classmate:1.5.1' sources to '/tmp/ort-ScanCode16026999469159078050/Maven/com.fasterxml/classmate/1.5.1' from VCS...
[2022-09-30T15:33:34.121Z] 15:33:31.601 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=ssh://git@github.com/FasterXML/java-classmate.git, revision=classmate-1.5.1, path=). Original was VcsInfo(type=Git, url=git@github.com:FasterXML/java-classmate.git, revision=classmate-1.5.1, path=).
[2022-09-30T15:33:34.121Z] 15:33:31.601 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
[2022-09-30T15:33:34.121Z] 15:33:31.610 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Adding Git revision 'classmate-1.5.1' (taken from package metadata) as a candidate.
[2022-09-30T15:33:34.381Z] 15:33:34.286 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package 'classmate' and version '1.5.1' found: IOException: Unable to list the remote tags.
[2022-09-30T15:33:34.381Z] Caused by: TransportException: ssh://git@github.com/FasterXML/java-classmate.git: Auth fail
[2022-09-30T15:33:34.381Z]     Caused by: TransportException: ssh://git@github.com/FasterXML/java-classmate.git: Auth fail
[2022-09-30T15:33:34.381Z]         Caused by: JSchException: Auth fail
[2022-09-30T15:33:34.381Z] 15:33:34.287 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Trying revision candidate 'classmate-1.5.1' (1 of 1)...
[2022-09-30T15:33:34.381Z] 15:33:34.287 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying to fetch only revision 'classmate-1.5.1' with depth limited to 50.
[2022-09-30T15:33:34.381Z] 15:33:34.292 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git fetch --depth 50 origin classmate-1.5.1' in '/tmp/ort-ScanCode16026999469159078050/Maven/com.fasterxml/classmate/1.5.1'...
[2022-09-30T15:33:34.382Z] 15:33:34.319 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch only revision 'classmate-1.5.1': IOException: Running 'git fetch --depth 50 origin classmate-1.5.1' in '/tmp/ort-ScanCode16026999469159078050/Maven/com.fasterxml/classmate/1.5.1' failed with exit code 128:
[2022-09-30T15:33:34.382Z] No user exists for uid 1000
[2022-09-30T15:33:34.382Z] fatal: Could not read from remote repository.
[2022-09-30T15:33:34.382Z] 
[2022-09-30T15:33:34.382Z] Please make sure you have the correct access rights
[2022-09-30T15:33:34.382Z] and the repository exists.

Any idea ?

sschuberth commented 2 years ago

Any idea ?

The Dockerfile has been completely rewritten. If you run into trouble, you might want to use docker/legacy/Dockerfile as a work-around for the time being.

tsteenbe commented 2 years ago

@fb33 could you post the exact series of Docker commands you are using to do a ORT scan of a public (open source) project. Looking for "steps to reproduce" the issue.

heliocastro commented 2 years ago

@fb33 How ( and where ) are you storing the ssh credentials ? The new container runs as a regular user. If you build it by default, the container user is ort, so the credentials need to be there.

Then i do recommend you try on the docker runtime do something similar like this:

docker run --mount type=bind,source="$(pwd)"/.ssh,target=/home/ort/.ssh,readonly ort

This of course is the most basic bind type, you can be more restrict if your environment need, like labeling, selinux, etc..

fb33 commented 2 years ago

Hi,

The implementation is a bit complexe due to the our CI process. ORT usage is drived by a jenkins pipeline in a docker in docker context. But I can resume it, in few steps :

1/ launch ORT container : docker run -t -d --entrypoint -e DOCKER_DAEMON_HOST=**** -v /home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/config1233094375676195490tmp:/etc/passwd -u ort:ort --cpus=3 -v$HOME/.ssh:/ort/.ssh -w /project -v /home/jenkins/agent/workspace/_oss/audit-ort-manually/project:/project -v /home/jenkins/agent/workspace/_oss/audit-ort-manually/ort_home:/ort/.ort -v /home/jenkins/agent/workspace/_oss/audit-ort-manually/npm_logs:/ort/.npm/_logs -e ORT_HTTP_USERNAME=**** -e ORT_HTTP_PASSWORD=**** -e JAVA_OPTS=-Xms512M -Xmx10g -XX:MaxMetaspaceSize=6g -Dkotlinx.coroutines.io.parallelism=10 -Dkotlin.daemon.jvm.options='-Xmx8g' -Dsvnkit.ssh2.key=/ort/.ssh/id_rsa -Dsvnkit.ssh2.username=**** -w ******** -e SVN_PASSWORD=**** -e SVN_USER=**** -e WORKSPACE=/home/jenkins/agent/workspace/_oss/audit-ort-manually docker.docker-registry.lectra.com/ort-lectra:node16.14.2_npm8.5.0_scancode30.1.0-TEST cat

2/ connect in container and prepare ssh context : (here some logs)

Running ssh-add (command line suppressed)
Identity added: /home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/private_key_16533079177528635265.key 
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/private_key_5471863660807846091.key 
[ssh-agent] Started.
[Pipeline] {
[Pipeline] configFileProvider
provisioning config files...
copy managed file [eu_autobuild_svc_git_config] to file:/home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/config6445647360620459756tmp
copy managed file [known_hosts] to file:/home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/config9475434430233463539tmp
[Pipeline] {
[Pipeline] sh
mkdir -p /home/ort/.ssh
[Pipeline] sh
cp /home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/config9475434430233463539tmp /home/ort/.ssh/known_hosts
[Pipeline] sh
cp /home/jenkins/agent/workspace/_oss/audit-ort-manually/project@tmp/config6445647360620459756tmp /home/ort/.gitconfig

3/ exec ORT command /usr/bin/ort --info --config /project/ort.conf scan --skip-excluded -f JSON -i /project/reports/analyzer/analyzer-result.yml -o /project/reports/scanner

heliocastro commented 2 years ago

Here:

-u ort:ort --cpus=3 -v$HOME/.ssh:/ort/.ssh

First -u is unecessary unless you are running with a complete different user Second, should not be /home/ort/.ssh ? Is mounting /ort/.ssh

fb33 commented 2 years ago

@heliocastro :+1: You are right! my bad, I quickly replace root by ort... I'll test with the good folder

fb33 commented 2 years ago

So, I go a step further. But now I've this error:

14:39:11  12:39:08.560 [main] INFO  org.ossreviewtoolkit.scanner.PathScanner - Finished scanning 'Maven:com.auth0:java-jwt:3.8.1' in thread 'main' (1 of 79).
14:39:11  12:39:08.561 [main] INFO  org.ossreviewtoolkit.scanner.PathScanner - Scanning 'Maven:com.fasterxml:classmate:1.5.1' in thread 'main' (2 of 79)
14:39:11  12:39:08.562 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'Maven:com.fasterxml:classmate:1.5.1' sources to '/tmp/ort-ScanCode5781155196184446427/Maven/com.fasterxml/classmate/1.5.1' from VCS...
14:39:11  12:39:08.564 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=ssh://git@github.com/FasterXML/java-classmate.git, revision=classmate-1.5.1, path=). Original was VcsInfo(type=Git, url=git@github.com:FasterXML/java-classmate.git, revision=classmate-1.5.1, path=).
14:39:11  12:39:08.564 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
14:39:11  12:39:08.572 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Adding Git revision 'classmate-1.5.1' (taken from package metadata) as a candidate.
14:39:11  12:39:11.398 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package 'classmate' and version '1.5.1' found: IOException: Unable to list the remote tags.
14:39:11  Caused by: TransportException: ssh://git@github.com/FasterXML/java-classmate.git: Auth fail
14:39:11      Caused by: TransportException: ssh://git@github.com/FasterXML/java-classmate.git: Auth fail
14:39:11          Caused by: JSchException: Auth fail
14:39:11  12:39:11.399 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Trying revision candidate 'classmate-1.5.1' (1 of 1)...
14:39:11  12:39:11.399 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying to fetch only revision 'classmate-1.5.1' with depth limited to 50.
14:39:11  12:39:11.404 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git fetch --depth 50 origin classmate-1.5.1' in '/tmp/ort-ScanCode5781155196184446427/Maven/com.fasterxml/classmate/1.5.1'...
14:39:12  12:39:12.301 [main] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch only revision 'classmate-1.5.1': IOException: Running 'git fetch --depth 50 origin classmate-1.5.1' in '/tmp/ort-ScanCode5781155196184446427/Maven/com.fasterxml/classmate/1.5.1' failed with exit code 128:
14:39:12  git@github.com: Permission denied (publickey).
14:39:12  fatal: Could not read from remote repository.
14:39:12  
14:39:12  Please make sure you have the correct access rights
14:39:12  and the repository exists.

Ort Git try to clone the project with SSH, but github seems to authorize clone only via https or git cli on "anonymous mode"

I've got the same error, if I try to clone the project on my ubuntu via ssh.

heliocastro commented 2 years ago

@fb33 Can you please paste the very first lines of ort run ( the ones where the ASCII ORT appears ? I'm pretty sure that ort is running with a complete different user then not finding yours .ssh files

fb33 commented 2 years ago

here are the lines : 

14:38:34  ________ _____________________
14:38:34  \_____  \\______   \__    ___/ the OSS Review Toolkit, version DOCKER-SNAPSHOT.
14:38:34   /   |   \|       _/ |    |
14:38:34  /    |    \    |   \ |    |    Running 'scan' under Java 11.0.16.1 on Linux with
14:38:34  \_______  /____|_  / |____|    3 CPUs and a maximum of 10240 MiB of memory.
14:38:34          \/       \/
14:38:34  Environment variables:
14:38:34  ORT_CONFIG_DIR = /home/ort/.ort/config
14:38:34  ORT_DATA_DIR = /home/ort/.ort
14:38:34  JAVA_HOME = /opt/java/openjdk
14:38:34  ANDROID_HOME = /opt/android-sdk
fb33 commented 2 years ago

I think, with my previous version of ORT docker image (build in august), the VCS url was with HTTPS and not with git SSH.

heliocastro commented 2 years ago

If was https, and not ssh, and you are behind proxy, that would explain most of everything ssh clones over proxy environment need something like corskscrew Good example: https://gist.github.com/ozbillwang/005bd1dfc597a2f3a00148834ad3e551

fb33 commented 2 years ago

Since 4c57907 with https://github.com/oss-review-toolkit/ort/blob/4c57907f5f3c95f229f824c8cd802ce0efdf660b/utils/ort/src/main/kotlin/Utils.kt#L181-L183 it should fix my problem ! :)

sschuberth commented 2 years ago

it should fix my problem ! :)

Feel free to close this issue then after you were able to verify the fix 😉

fb33 commented 2 years ago

Hi @sschuberth and @fviernau

I think a case is missing in the process to "Stop using the unauthenticated Git protocol in VCS processed"

I made a PR : https://github.com/oss-review-toolkit/ort/pull/5951

sschuberth commented 2 years ago

I made a PR : #5951

I've commented over there.

schvvarzekatze commented 2 years ago

I still get the ssh error in the scanner step for all repositories cloned by ssh. I reproduced it for one sample dependency:

11:48:18.202 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Adding Git revision 'jasypt-1.9.3' (taken from package metadata) as a candidate.
11:48:20.375 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package 'jasypt' and version '1.9.3' found: IOException: Unable to list the remote tags.
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
        Caused by: JSchException: Auth fail
11:48:20.378 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver - Trying revision candidate 'jasypt-1.9.3' (1 of 1).
11:48:20.388 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Trying to fetch only revision 'jasypt-1.9.3' with depth limited to 50.
11:48:28.833 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch only revision 'jasypt-1.9.3': TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: JSchException: Auth fail
11:48:28.833 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Falling back to fetching all refs with depth limited to 50.
11:48:30.613 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch with only a depth of 50: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: JSchException: Auth fail
11:48:30.614 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Falling back to fetch everything including tags.
11:48:32.375 [DefaultDispatcher-worker-3] WARN  org.ossreviewtoolkit.downloader.vcs.Git - Failed to fetch everything: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: JSchException: Auth fail
11:48:32.376 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver - Could not resolve revision candidate 'jasypt-1.9.3': TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: JSchException: Auth fail
11:48:32.442 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver - Could not resolve VCS for 'Maven:org.jasypt:jasypt:1.9.3': IOException: Could not resolve revision for package 'Maven:org.jasypt:jasypt:1.9.3' with VcsInfo(type=Git, url=ssh://git@github.com/jasypt/jasypt.git, revision=jasypt-1.9.3, path=):
        Could not resolve revision candidate 'jasypt-1.9.3': TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
Caused by: TransportException: ssh://git@github.com/jasypt/jasypt.git: Auth fail
    Caused by: JSchException: Auth fail

When I change the vcs_processed to

          type: "Git"
          url: "https://github.com/jasypt/jasypt.git"
          revision: "jasypt-1.9.3"
          path: ""

the scan is without the error. I used the docker image of ort updated today.

What could be a quick fix in my case to succesfully scan all repositories cloned by ssh?

Thank you very much.

sschuberth commented 2 years ago

What could be a quick fix in my case to succesfully scan all repositories cloned by ssh?

In contrast to HTTP(S), you cannot clone unauthenticated via SSH. That means, you have to provide credentials when cloning via SSH. So, when cloning via SSH from GitHub, you need to have an account on GitHub that you can use and authenticate with. And ideally, the credentials should be provided by an SSH agent; then JSch used by JGit will pick them up.

limod commented 2 years ago

Hi! I also had problemes with ssh + auth fail and added a SSH Key for our private repository via the SSH-Agent. While testing i mounted my local ssh-agent into docker: -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent

This resolved the Auth fail exception (Thanks!) but now i get the following error:

13:40:53.725 [DefaultDispatcher-worker-50] INFO  org.ossreviewtoolkit.downloader.vcs.Git - Could not fetch only revision '2a872f23d36cfb2c44b1c725c44a9ad4b37f6838': TransportException: ssh://git@git.xx.com/xx/xx.git: remote hung up unexpectedly
Caused by: TransportException: ssh://git@git.xx.com/xx/xx.git: remote hung up unexpectedly
    Caused by: TranslationBundleLoadingException: Loading of translation bundle failed for [org.eclipse.jgit.internal.transport.ssh.jsch.JSchText, en_US]
        Caused by: MissingResourceException: Can't find bundle for base name org.eclipse.jgit.internal.transport.ssh.jsch.JSchText, locale en_US

Do you have any idea what causes this error?

sschuberth commented 2 years ago

Loading of translation bundle failed for [org.eclipse.jgit.internal.transport.ssh.jsch.JSchText, en_US]

See this for some background information. @limod, do you use a custom ORT distribution by any chance, like a custom-built Fat JAR?

limod commented 2 years ago

@sschuberth I do not use a custom ORT distribution. I cloned the repository (on 19.10.22) and executed docker build -t ort .

sschuberth commented 2 years ago

@limod despite the ordering in the stacktrace, it seems that remote hung up unexpectedly is the root cause, and Can't find bundle for base name is just a follow-up error as JGit fails to find a translation for some JSch-related message. I've prepared a fix to JGit for that, but like I said, that's likely not the root cause.

limod commented 2 years ago

Sorry for the delay, the remote hung up unexpectedly has resolved, seemed that there was a real issue while testing with the repository. I ended in a Invalid Key Exception but there is already an issue for this error: #6024

sschuberth commented 2 years ago

I ended in a Invalid Key Exception but there is already an issue for this error: #6024

... and a potential solution 😉 @limod could you also please give https://github.com/oss-review-toolkit/ort/pull/6030 a try in this regard?

sschuberth commented 2 years ago

@limod could you also please give #6030 a try in this regard?

I'm confident that the merged #6030 solves this issue. If not, feel free to reopen.

sschuberth commented 1 year ago

@limod despite the ordering in the stacktrace, it seems that remote hung up unexpectedly is the root cause, and Can't find bundle for base name is just a follow-up error as JGit fails to find a translation for some JSch-related message. I've prepared a fix to JGit for that, but like I said, that's likely not the root cause.

FYI, this specific issue has been fixed with https://github.com/oss-review-toolkit/ort/pull/6148.