Analyzer fails to authenticate with Artifactory when downloading artifacts (http-401)

[software-testing-professional]

Hi there,

I'm having trouble to authenticate against Artifactory when running ort analyze.

Credendials for Artifactory (username / password) are provided via .netrc file and Maven settings.xml. But download attempts always result in a http-401 unauthorized.

Downloading these artifacts via cURL works fine, when I provide username / password. So it seems that the configuration on Artifactory side is fine.

Did I miss some configuration? What can I do to solve this issue? I appreciate your help! :-)

08:31:50.127 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Remote location for 'external.c:openssl:jar:sources:1.1.1n': external/c/openssl/1.1.1n/openssl-1.1.1n-sources.jar
08:31:50.127 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultTransporterProvider - Using transporter HttpTransporter with priority 5.0 for https://artifactory.*****.com/artifactory/its-external
08:31:50.127 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector BasicRepositoryConnector with priority 0.0 for https://artifactory.*****.com/artifactory/its-external
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://artifactory.*****.com:443][total/ available: 2; route allocated: 1 of 50; total allocated: 2 of 100]
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {s}->https://artifactory.*****.com:443][total/ available: 1; route allocated: 1 of 50; total allocated: 2 of 100]
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: set socket timeout to 0
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: set socket timeout to 1800000
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request HEAD /artifactory/its-external/external/c/openssl/1.1.1n/openssl-1.1.1n-sources.jar HTTP/1.1
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
08:31:50.128 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
08:31:50.154 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
08:31:50.154 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.auth.HttpAuthenticator - Authentication required
08:31:50.154 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.auth.HttpAuthenticator - artifactory.*****.com:443 requested authentication
08:31:50.154 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Authentication schemes in the order of preference: [Negotiate, Kerberos, NTLM, CredSSP, Digest, Basic]
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Negotiate authentication scheme not available
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Kerberos authentication scheme not available
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for NTLM authentication scheme not available
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for CredSSP authentication scheme not available
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Digest authentication scheme not available
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {s}->https://artifactory.*****.com:[443](https://gitlab.*****.com/mocca/oss-review-toolkit/-/jobs/5359208#L443)] can be kept alive indefinitely
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: set socket timeout to 0
08:31:50.155 [DefaultDispatcher-worker-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {s}->https://artifactory.*****.com:443]/[total available: 2; route allocated: 1 of 50; total allocated: 2 of 100]
08:31:50.155 [DefaultDispatcher-worker-3] WARN  org.apache.http.client.protocol.ResponseProcessCookies - Invalid cookie header: "Set-Cookie: AWSALBTG=wXE1osuo2pSAo+9NrSYXQ0EgQ2RySOklfj2QVhTXb2XX8HUxx5SaEIuyM0g+1x0pNOGyJRicNpJu9twEyDD33tSMSP9Y1ErNosFyl01UlYBi14GuJKNcVbUymYQIuzH67Osj5QbGasz4uEtYTYXYnLmrRmZgASaGHeoNH6JETQIxu72H***=; Expires=Wed, 06 Jul 2022 08:31:50 GMT; Path=/". Invalid 'expires' attribute: Wed, 06 Jul 2022 08:31:50 GMT
08:31:50.155 [DefaultDispatcher-worker-3] WARN  org.apache.http.client.protocol.ResponseProcessCookies - Invalid cookie header: "Set-Cookie: AWSALBTGCORS=wXE1osuo2pSAo+9NrSYXQ0EgQ2RySOklfj2QVhTXb2XX8HUxx5SaEIuyM0g+1x0pNOGyJRicNpJu9twEyDD33tSMSP9Y1ErNosFyl01UlYBi14GuJKNcVbUymYQIuzH67Osj5QbGasz4uEtYTYXYnLmrRmZgASaGHeoNH6JETQIxu72H***=; Expires=Wed, 06 Jul 2022 08:31:50 GMT; Path=/; SameSite=None; Secure". Invalid 'expires' attribute: Wed, 06 Jul 2022 08:31:50 GMT
08:31:50.156 [DefaultDispatcher-worker-3] WARN  org.apache.http.client.protocol.ResponseProcessCookies - Invalid cookie header: "Set-Cookie: AWSALB=Aj+xDryH81fx1LpTd/dzakMUkCUkt999yNNXJF8kysW7vCWHmFINz4B1EKXdDg+QDsp61KiKbnP3qvBQP21oJMEyTFPnDTQRNl/KXxIoojVo0DDjX7niHOIXhG4a; Expires=Wed, 06 Jul 2022 08:31:50 GMT; Path=/". Invalid 'expires' attribute: Wed, 06 Jul 2022 08:31:50 GMT
08:31:50.156 [DefaultDispatcher-worker-3] WARN  org.apache.http.client.protocol.ResponseProcessCookies - Invalid cookie header: "Set-Cookie: AWSALBCORS=Aj+xDryH81fx1LpTd/dzakMUkCUkt999yNNXJF8kysW7vCWHmFINz4B1EKXdDg+QDsp61KiKbnP3qvBQP21oJMEyTFPnDTQRNl/KXxIoojVo0DDjX7niHOIXh***; Expires=Wed, 06 Jul 2022 08:31:50 GMT; Path=/; SameSite=None; Secure". Invalid 'expires' attribute: Wed, 06 Jul 2022 08:31:50 GMT
08:31:50.156 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Transfer failed: GET_EXISTENCE FAILED https://artifactory.*****.com/artifactory/its-external/external/c/openssl/1.1.1n/openssl-1.1.1n-sources.jar <> /root/.m2/repository/external/c/openssl/1.1.1n/openssl-1.1.1n-sources.jar
08:31:50.156 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Could not find 'external.c:openssl:jar:sources:1.1.1n' in 'https://artifactory.*****.com/artifactory/its-external (https://artifactory.*****.com/artifactory/its-external, default, releases+snapshots)': ArtifactTransferException: Could not transfer artifact external.c:openssl:jar:sources:1.1.1n from/to https://artifactory.*****.com/artifactory/its-external (https://artifactory.*****.com/artifactory/its-external): status code: 401, reason phrase:  (401)
Caused by: HttpResponseException: status code: 401, reason phrase:  (401)
08:31:50.156 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'external.c:openssl:jar:sources:1.1.1n' in any of [https://repo.maven.apache.org/maven2, https://artifactory.*****.com/artifactory/its-external].
08:31:50.156 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Writing empty remote artifact for 'external.c:openssl:jar:sources:1.1.1n' to disk cache.

(i) Some information like URLs and header information have been obfuscated with ***

[sschuberth]

Downloading these artifacts via cURL works fine, when I provide username / password.

Are you using exactly the same URL as the ORT analyzer for this check? Because I recall there was a subtle difference in URLs that artifactory shows, which sometimes contain "api" as part of the path, and sometimes not. An IIRC in one of the cases the user's API key instead of e.g. an AD password needs to be used.

Maybe @MarcelBochtler remembers some more details.

[software-testing-professional]

Yes, the download link shown in Artifactory's native file browser matches the one used by ORT. This link was used in my cURL test.

I tried both username / token and username / API key in the settings.xml. Each time it resulted in a 401, wen ORT analyze was run.

[software-testing-professional]

I did some more investigation and found out, that the download requests performed by ORT do not reach Artifactory at all. The access log of Artifactory does not show any download attempts at this time. Could this be related to the invalid cookie messages? And the AWS application load balancer blocks all download requests because missing request attributes?

[sschuberth]

Could this be related to the invalid cookie messages?

Could be. Looks like this is more or less a know issue with the Apache HTTP client that the Maven resolver uses. The answer on SO has a solution on how to fix this for the Apache HTTP client directly, but we'd yet need to find out how to fix this for the Maven resolver / the client that the resolver uses.

[software-testing-professional]

Meanwhile we tried some configuration on the loadbalancer. But without effect.

The credentials used for these type of downloads (executed by MavenSupport class) are taken either from .netrc or ENV variables (ORT_HTTP_USERNAME and ORT_HTTP_PASSWORD), right?

Requests on Artifactory side look like

2022-07-01T10:17:55.773Z|2936dde2010f2f4e|79.219.239.209|non_authenticated_user|HEAD|/its-external/external/c/openvpn/2.5.1/openvpn-2.5.1.tar.xz|401|-1|0|1|Apache-Maven/3.8.5 (Java 11.0.15; Linux 5.13.0-44-generic)

Is there any way to add a custom request header to these requests? Might be worth a try. I'm using Artifactory as httpStorageBackend for the scanner, with an Authorization header that holds a bearer token. This works fine.

[software-testing-professional]

Might our issues be related to this? https://github.com/oss-review-toolkit/ort/blob/main/analyzer/src/main/kotlin/managers/Gradle.kt#L225

Because we are only having authentication errors with Gradle builds. And Artifactory says non_authenticated_user is requesting files.

[mawl]

@sschuberth: Same issue here. As you can see, username and password is shown in the log:

7:14:27.367 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultTransporterProvider - Using transporter HttpTransporter with priority 5.0 for https://repo.acme.de/artifactory/repo
07:14:27.367 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector BasicRepositoryConnector with priority 0.0 for https://repo.acme.de/artifactory/repo with username=SomeUser01, password=***
07:14:27.374 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultTransporterProvider - Using transporter HttpTransporter with priority 5.0 for https://repo.maven.apache.org/maven2
07:14:27.375 [DefaultDispatcher-worker-3] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
07:14:27.379 [DefaultDispatcher-worker-3] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - There have been issues building the Maven project models, this could lead to errors during dependency analysis: ProjectBuildingException: Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for com.acme.pom:acme-java-pom:7.2.0: Could not transfer artifact com.acme.pom:acme-root-pom:pom:4.0.4 from/to artifactory (https://repo.acme.de/artifactory/repo): /home/ort/.m2/repository/com/acme/pom/acme-root-pom/4.0.4/acme-root-pom-4.0.4.pom.part.lock (No such file or directory) and 'parent.relativePath' points at wrong local POM @ line 14, column 10

[mawl]

Issue is now fixed for us - bei using

# fix file permissions
RUN chown -R ort:ort ${HOME}

# use non-root user at runtime
USER ort

in our custom Dockerfile, after COPY of the maven settings.xml.

All new files and directories are created with a UID and GID of 0, unless the optional --chown flag specifies a given username, groupname, or UID/GID combination to request specific ownership of the copied content

https://docs.docker.com/engine/reference/builder/#copy

[sschuberth]

Interesting. Do you understand why that change fixed the issue? At first I thought that settings.xml wasn't found due to looking in the wrong user home. But then I remembered that username and password appeared in the log, so apparently the settings.xml was found.

[mawl]

I suspect maven has no permissions to create files under /home/ort/.m2/repository - but has read permissions on the settings.xml file.

[sschuberth]

Might our issues be related to this? https://github.com/oss-review-toolkit/ort/blob/main/analyzer/src/main/kotlin/managers/Gradle.kt#L225

@software-testing-professional Unfortunately, you didn't use a permalink, and by now that line points to if (!initScriptFile.delete()) {, which I believe is unrelated. Would you mind repeating which line you had in mind?

[sschuberth]

I suspect maven has no permissions to create files under /home/ort/.m2/repository - but has read permissions on the settings.xml file.

@mawl you mentioned that you're using your own "custom Dockerfile". So, can you confirm that there is nothing to fix in any of our two Dockerfiles? Otherwise, feel free to create a PR.

[mawl]

@sschuberth: We first build your image based on https://github.com/oss-review-toolkit/ort/blob/main/Dockerfile and use it in the FROMstatement of our Dockerfile.

There we install some additional tools like cyclonedx-cli or hashicorp vault cli and copying package manager settings into it. So no, everything is fine with your files. We only had to include the switch back from USER root to USER ort after our installations.

[software-testing-professional]

@sschuberth Sorry for that. I found a code comment here: https://github.com/oss-review-toolkit/ort/blob/aa942fa62dd7f055e2f27f51831dc66cf77e55c8/analyzer/src/main/kotlin/managers/Gradle.kt#L230

// TODO: Also handle authentication and snapshot policy.

This led me to the assumption that something related to authentication might still be missing.

[sschuberth]

Sorry for that.

No worries. Thanks for being quick in posting an update!

This led me to the assumption that something related to authentication might still be missing.

I'm currently unsure whether this old comment of @mnonnenmacher from 2017 is still valid.

@software-testing-professional looks like you're running ORT from a Docker image. Could you also try running ORT natively with the same configuration to rule out any Docker-related issues?

[software-testing-professional]

@sschuberth Sorry, I'm a bit late with my answer. ;-) I also tried to use ORT natively. With same results. Normally I use the ORT Docker image and mount all the configuration (settings.xml, .netrc, etc.) into the container.

The repository configuration is done via Gradle. And the Artifactory repositories are configured as

maven {
          credentials {
              username lUsr
              password lPwd
          }
          url lUrl

Although everything was configured, Artifactory only got requests from an "unauthenticated" user. Because the requested resource was part of the Gradle project, these requests definitely came from ORT.

But: I did some more testing, and could solve the authentication issue by adding this to gradle.properties and mounting it into the container:

systemProp.http.proxyUser=<user>
systemProp.http.proxyPassword=<pw>

So currently the Gradle authentication still does not work. But solved via proxy authentication.

[sschuberth]

@software-testing-professional would you mind checking whether the current version of ORT that includes https://github.com/oss-review-toolkit/ort/pull/6498 fixes the issue for you?

[software-testing-professional]

@sschuberth Yes, I'll be able to try that next week from Wednesday on. :+1:

[software-testing-professional]

I can confirm that the scanner does not freeze anymore, if ClearlyDefined is defined as storageReader (together with artifactoryStorage).

    storageReaders: [
      "clearlyDefined",
      "artifactoryStorage"
    ]

...
08:30:57.397 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:io.swagger.core.v3:swagger-core:2.2.7' from ClearlyDefinedStorage in 694.389us.
08:30:57.979 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 1 scan result(s) for 'Maven:org.apache.tomcat.embed:tomcat-embed-el:10.1.4' from ClearlyDefinedStorage in 582.341739ms.
08:30:57.981 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:net.logstash.logback:logstash-logback-encoder:7.2' from ClearlyDefinedStorage in 1.274609ms.
08:30:57.982 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:org.springframework:spring-test:6.0.3' from ClearlyDefinedStorage in 890.483us.
...

Tested with Docker-based ORT, built from commit 19c89ff9a.

Thank you for fixing it!!! :smiley:

[sschuberth]

I can confirm that the scanner does not freeze anymore, if ClearlyDefined is defined as storageReader

Hmm, this sounds a bit as if you're confusing this issue with https://github.com/oss-review-toolkit/ort/issues/4540 😉

But what about this:

I'm having trouble to authenticate against Artifactory when running ort analyze.

[software-testing-professional]

Ah right - that happens if you have too many open tabs. :laughing: I'll move that answer over to the other issue.

Regarding authentication, this also works with the Docker-based ORT built from commit https://github.com/oss-review-toolkit/ort/commit/19c89ff9a0a7aa2a52d85c82fd477531da1ecf3d.

[sschuberth]

Regarding authentication, this also works with the Docker-based ORT built from commit 19c89ff.

Great, thanks for confirming, so I'll be closing this!