Galasa needs to load test artifacts from a password-protected maven repo

[eamansour]

Story

As a galasa user I want to be able to run tests on an ecosystem which are stored in a password-protected maven repository, so that I know only authorised people can see my test material.

Background

Currently, Galasa does not support loading test material from a password-protected maven repository.

Current behaviour if you try it

After deploying an OBR to a Maven repository requiring users to be logged in to deploy artifacts to, and then running tests via an ecosystem's test stream, the Maven repository returns a 401 as the tests fail to pull the deployed OBR (see error below). This happens even if the correct credentials are provided in your settings.xml file.

java.io.IOException: Server returned HTTP response code: 401 for URL: https://host/artifactory/maven-repository/path/to/OBR/0.1.0-SNAPSHOT/maven-metadata.xml

Steps to reproduce

1. Start a Galasa ecosystem
2. Deploy an OBR and a test to a Maven repository that requires authentication
3. Create a test stream containing the deployed test
4. Create a portfolio and submit the test to the ecosystem
5. View the test's run log, which should show the 401 error.

Expected behaviour

The OBR should be pulled in successfully when running tests on the ecosystem.

Current workaround

One workaround is to just set up a Maven repository that doesn't require authentication, which is not ideal.

Proposed solution

• The credentials for the maven artefact repo are stored in the creds namespace in CPS properties.
• The stream has an extra (optional) field, indicating which credential property is used to access the maven repo
• The test runner is passed the credential secret by the framework
• The test runner sets the credentials into the OSGi framework (somehow) so they get used when the maven artefacts are loaded.

Tasks

• [ ] Spike story to investigate

Main story...

• [ ] Get hold of a usable artifactory to use to test against.
• [ ] Change stream code to house the extra parameter
• [ ] ? to get local drawing from the creds store ?
• [ ] Docs
• [ ] Integration testing

[techcobweb]

This is a problem reported by @Thomas Slattery and Daniel Donnelly of IBM UK CICS team.

[techcobweb]

Proposed solution outline:

• The credentials for the maven artifact repo are stored in the creds namespace in CPS properties.
• The stream has an extra (optional) field, indicating which credential property is used to access the maven repo
• The test runner is passed the credential secret by the framework
• The test runner sets the credentials into the OSGi framework (somehow) so they get used when the maven artifacts are loaded.

[techcobweb]

Configure the JVM with a URL handler...

Java has this call : URLConnection.setContentHandlerFactory - Java

~~If we set this up to call our custom URL handler, will attempts to load bundles from Felix pass through it ? This potential avenue isn't my favourite. See below...~~ This call is about capturing in-board HTTP traffic and handling it, not out-bound.

Configure Felix with a URL handler...

~~felix.service.urlhandlers is a property which can be passed to Felix which causes the framework to "activate the URL Handlers service for the framework instance; the default value is true. Activating the URL Handlers service will result in the URL.setURLStreamHandlerFactory() and URLConnection.setContentHandlerFactory() being called."~~ (See https://felix.apache.org/documentation/subprojects/apache-felix-framework/apache-felix-framework-configuration-properties.html#_framework_configuration_properties)

Might it be possible to interject on outgoing traffic, and from the URL look up some credentials to add to the basic auth header in the request ?

The URL handler is about processing in-bound HTTP traffic, so is not applicable here.

Is there a way of registering a handler with the OSGi framework which would be called when outbound traffic was being sent ?

This would be my preferred approach, as any custom handler would be executing within the OSGi environment, and any of the Galasa framework calls to get the credentials would be reachable from there.

Forward proxy idea

I think this is a non-starter, but details just in case, to show what I was reading...

Felix can be configured to point to a web proxy. See https://felix.apache.org/documentation/subprojects/apache-felix-framework/apache-felix-framework-usage-documentation.html#_web_proxy_issues_when_installing_bundles

The properties can be set by galasa-parent/galasa-boot/src/main/java/dev/galasa/boot/felix/FelixFramework.java (See the buildFramework method)

These can specify basic auth properties which will be sent to the proxy server. And the proxy server could then pass them on to the secure maven site/artifactory.

NGinX can be configured as a passthrough forward proxy server I believe.

There is possibly an option of injecting the password using the forward proxy server like in this article: https://serverfault.com/questions/230749/how-to-use-nginx-to-proxy-to-a-host-requiring-authentication

[techcobweb]

Forget the above methods. The framework does all the comms to load bundles. Look at the fetchReleaseArtifact(...) method in galasa-parent/dev.galasa.framework.maven.repository/src/main/java/dev/galasa/framework/maven/repository/internal/GalasaMavenUrlHandlerService.java

There you will see this:

connection.setConnectTimeout(connectionTimeoutMilliSecs);
connection.setReadTimeout(readTimeoutMilliSecs);
...   
connection.connect();

I think we need to do this: connection.addRequestProperty("Authorization","Basic "+base64(userid+":"+password)); or similar to set the header. There seem to be 3-4 places where this needs doing ? Why ?

The next question is how to get the userids and passwords into this code in the first place.

On a server, these details need to come from the creds store. So, for the stream we need to set a secret id to use.

How would that work locally ? The runs submit local command must supply it. Perhaps with a new flag --mavenauth{userid}:{password} ?

[techcobweb]

A duplicate of this story was raised here: [As Galasa Admin I need an option to pass credentials to galasa ecosystem to pull OBRs from external repository like JFrog

1907](https://github.com/galasa-dev/projectmanagement/issues/1907)

By @haripra86

At present there is no way to pass credentials to Galasa ecosystem so it can pull OBR from private artifactories like JFrog etc. We need this option so we can host code on safe repositories.

I see 401 auth response while submitting tests.

java.io.IOException: Server returned HTTP response code: 401 for URL: https://na.xxxxxx.xxxx/zapm-tests/ibm.zaiops.zapm.tests.obr/0.0.1-SNAPSHOT/maven-metadata.xml

This needs to be prioritised as it will compell admin to create and maintain a unsecured artifactories just to make galasa system work while IBM provides more secured artifactories.