Proxy error via container #295

Closed BrunoVazCosta closed 2 years ago

APIM-CLI version

1.12.1

API-Management version

7.7.20210330

Question

In our GitLab CI/CD we're going to use the APIM CLI containerized. The container image has the cli, the certificate and an entrypoint.sh file. In the CLI's /conf folder, on env.properties, we set the proxy settings. The certificate is imported via keytool during the image creation before pushing to our registry. There's no communication issue between GitLab and the OpenShift where API Manager is installed. Although, when we run the cli we get an 407 error, which is Proxy Authentication Required. And we don't know what's missing to establish communication between the CLI x API Manager.

This is how we create the image

FROM [our-registry]/base/java/openjdk:11
COPY apim-cli-1.12.1.tar entrypoint.sh manager-ti.cer /
RUN keytool -v -importcert -alias apim -storepass apiadmin -noprompt -trustcacerts -keystore cacerts -file /manager-ti.cer && tar -xvf /apim-cli-1.12.1.tar && keytool -v -list -alias apim -keystore cacerts -storepass apiadmin
ENTRYPOINT ["/entrypoint.sh"]

This is the entrypoint.sh

#!/bin/bash

CLI=./apim-cli-1.12.1/scripts/apim.sh
echo "Using API-Manager: https://${INPUT_APIMHOSTNAME}:${INPUT_APIMPORT}"

apimCLIArgs="-h ${INPUT_APIMHOSTNAME} -port ${INPUT_APIMPORT} -u ${INPUT_APIMUSERNAME} -p ${INPUT_APIMPASSWORD}"

if ! [ -z ${INPUT_APIMCLICOMMAND} ];then
    echo "Calling APIM-CLI with given command: ${INPUT_APIMCLICOMMAND}"
    $CLI ${INPUT_APIMCLICOMMAND} ${apimCLIArgs} || exit 99;
fi

exit

That's the error message we get.

Hi @BrunoVazCosta ,

Your steps are looking good to me. I have tested with following parameters in conf/env.properties and it is working fine.

httpProxyHost=localhost httpProxyPort=3128

Hi @rathnapandi,

I guess the way we're importing the certificate may have some issue. I've tried both .pem and .cer with the chain provided by exporting certificate via web browser, but once the container starts and run the CLI, the 407 error occurs.

Does Axway have any info about how to properly create / configure the CLI in container?

Thanks!

Hi @BrunoVazCosta,

I will check internally and get back to you on container based on apim-cli.

Also, instead of changing cacerts file, we can create a truststore file, mount it and inject the location via java system properties.

-Djavax.net.ssl.trustStore=/opt/apimcli/certs/.jks -Djavax.net.ssl.trustStorePassword=

Hi @rathnapandi, sorry for still bothering you with this issue.

Do you have any info or manual of how to build an image with the CLI and its certificates in proper manner?

Hi @rathnapandi,

please, is there any containerized version of the cli? Perhaps a dockerfile sample?

I'm having a hard time investigating what's wrong with its configuration, because I'm still getting 407 as response.

Find below our dockerfile.

FROM [docker-registry.com.br]/base/java/openjdk:11 USER root COPY apim-cli-1.12.1.tar manager-ti.cer manager-ti-int.cer manager-ti-root.cer / RUN mv /manager-ti.cer /manager-ti-int.cer /manager-ti-root.cer /etc/pki/ca-trust/source/anchors/ \ && update-ca-trust extract \ && tar -xvf /apim-cli-1.12.1.tar \ && rm -f /apim-cli-1.12.1.tar COPY --chown=jboss:jboss entrypoint.sh / USER jboss ENTRYPOINT ["/entrypoint.sh"]

Thanks.

@BrunoVazCosta,

I am working on it, will get back to you later today.

Hi @BrunoVazCosta ,

Sample docker file located at https://github.com/Axway-API-Management-Plus/apim-cli/blob/develop/Dockerfile and instructions available on https://github.com/Axway-API-Management-Plus/apim-cli/wiki/9.4-Docker-Image.

We can create a truststore and add truststore specific system properties as JVM properties via the following command.

docker run -e "JAVA_TOOL_OPTIONS=-Dlog4j.configurationFile=/opt/apim-cli-1.12.1/conf/log4j2.xml -Djavax.net.ssl.trustStore=/opt/certs/custom.jks -Djavax.net.ssl.trustStorePassword=changeit" -e LOG_LEVEL=debug apimcli app get -u apiadmin -p changeme2 -h 172.17.0.1

Also, looks like CLI trusts all certificates - Reference https://github.com/Axway-API-Management-Plus/apim-cli/blob/develop/modules/apim-adapter/src/main/java/com/axway/apim/lib/utils/rest/APIMHttpClient.java

try { builder.loadTrustMaterial(null, new TrustAllStrategy());

        SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build(), new NoopHostnameVerifier());

Can you share the full error messages as well?

Hi @rathnapandi I've set the Java properties you've mentioned but still get 407

Hi @BrunoVazCosta Can you share the full stack trace / logs of apim cli and logs from api gateway too?

Hi @BrunoVazCosta,

Based on the http error code 407, we need to setup http proxy credentials with following variables via command line parameters or env.properties.

httpProxyUsername httpProxyPassword

Hi @BrunoVazCosta,

Based on the http error code 407, we need to setup http proxy credentials with following variables via command line parameters or env.properties.

httpProxyUsername httpProxyPassword

It's also set.

Hi @BrunoVazCosta Can you share the full stack trace / logs of apim cli and logs from api gateway too?

I'll see what I can get.

Hi @BrunoVazCosta ,

The apim cli http client code is not handling the proxy authentication flow correctly. The issue is fixed now.

Please test it with binary axway-apimcli-1.12.2-SNAPSHOT.tar.gz

Hi @rathnapandi, and thanks for digging into this issue. I've tested ignoring the PaxHeaders.X folder inside the compressed file, deploying apim-cli-1.12.2-SNAPSHOT.tar only. By doing that and running from inside the container I got the following error:

`---------------------------------------------------------------------------------------- API-Manager CLI: 1.12.2-SNAPSHOT

To report issues or get help, please visit: https://github.com/Axway-API-Management-Plus/apim-cli

Module: API - E X P O R T / U T I L S (1.12.2-SNAPSHOT)

967 DEBUG nvironmentProperties| Loaded environment properties from file: env.properties 1166 DEBUG APIManagerAdapter| Logging in with User: 'apiadmin' Jul 27, 2022 2:42:56 PM org.apache.http.impl.auth.HttpAuthenticator generateAuthResponse WARNING: NEGOTIATE authentication error: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm) Jul 27, 2022 2:42:56 PM org.apache.http.impl.auth.HttpAuthenticator generateAuthResponse WARNING: NTLM authentication error: Credentials cannot be used for NTLM authentication: org.apache.http.auth.UsernamePasswordCredentials 1775 WARN APIManagerAdapter| Login failed with statusCode: 407 ... Try again in 1000 milliseconds. (you may set -retryDelay ) 2829 ERROR AppException| Can't login to API-Manager https://manager-interno-api-ti.safra.com.br:443/api/portal/v1.4/login | Login finally failed with statusCode: 407 com.axway.apim.lib.errorHandling.AppException: Can't login to API-Manager https://manager-interno-api-ti.safra.com.br:443/api/portal/v1.4/login at com.axway.apim.adapter.APIManagerAdapter.loginToAPIManager(APIManagerAdapter.java:258) ~[apimcli-apim-adapter-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.adapter.APIManagerAdapter.(APIManagerAdapter.java:185) ~[apimcli-apim-adapter-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.adapter.APIManagerAdapter.getInstance(APIManagerAdapter.java:147) ~[apimcli-apim-adapter-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.APIExportApp.execute(APIExportApp.java:201) ~[apimcli-apis-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.APIExportApp.exportAPI(APIExportApp.java:72) ~[apimcli-apis-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.APIExportApp.exportAPI(APIExportApp.java:62) ~[apimcli-apis-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at com.axway.apim.cli.APIManagerCLI.run(APIManagerCLI.java:141) ~[apimcli-tool-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] at com.axway.apim.cli.APIManagerCLI.main(APIManagerCLI.java:77) ~[apimcli-tool-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] Caused by: com.axway.apim.lib.errorHandling.AppException: Login finally failed with statusCode: 407 at com.axway.apim.adapter.APIManagerAdapter.loginToAPIManager(APIManagerAdapter.java:241) ~[apimcli-apim-adapter-1.12.2-SNAPSHOT.jar:1.12.2-SNAPSHOT] ... 11 more`

Even setting those Java properties it gets the same error

@BrunoVazCosta The current APIM CLI implementation does not support authentication proxy using NTLM or kerberos. But we can add NTLM authentication if you validate the following code in your environment as I don't have setup to test it.

Edit username, password, and domain of proxy server

The code uses apache http client version 4.5.13.

package com.axway.apim.lib.utils.rest;

import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.NTCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustAllStrategy;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.DefaultProxyRoutePlanner;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;

public class ProxyAuth {

    public static void main(String[] args) throws IOException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
        SSLContextBuilder builder = new SSLContextBuilder();
        builder.loadTrustMaterial(null, new TrustAllStrategy());
        SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build(), new NoopHostnameVerifier());
        Registry<ConnectionSocketFactory> r = RegistryBuilder.<ConnectionSocketFactory>create()
                .register("https", sslsf)
                .register("http", PlainConnectionSocketFactory.INSTANCE)
                .build();
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(r);
        cm.setMaxTotal(5);
        cm.setDefaultMaxPerRoute(2);
        CredentialsProvider credsProvider = new BasicCredentialsProvider();
        credsProvider.setCredentials(
                new AuthScope("proxy.safra.com.br", 8080), // proxy hostname and port
                //new UsernamePasswordCredentials("admin", "changeit"));
                new NTCredentials("admin", "changeit", "", "domain")); // replace domain with NTLM domain

        HttpHost proxyHost = new HttpHost("proxy.safra.com.br", 8080); //
        HttpClientBuilder clientBuilder = HttpClients.custom()
                .setConnectionManager(cm);
        DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxyHost);
        clientBuilder.setRoutePlanner(routePlanner);
        RequestConfig config = RequestConfig.custom()
                .setProxy(proxyHost)
                .setRedirectsEnabled(true)
                .setMaxRedirects(5)
                .setConnectTimeout(100 * 1000)
                .setConnectionRequestTimeout(300 * 1000)
                .setSocketTimeout(300 * 1000)
                .build();
        clientBuilder.setDefaultCredentialsProvider(credsProvider);
        clientBuilder.setDefaultRequestConfig(config);
        CloseableHttpClient httpclient = clientBuilder.build();
        HttpGet httppost = new HttpGet("/home");
        httppost.setConfig(config);
        HttpHost target = new HttpHost("manager-interno-api-ti.safra.com.br", 443, "https");
        HttpResponse response = httpclient.execute(target, httppost);
        System.out.println("Return status code is " + response.getStatusLine().getStatusCode());
        System.out.println(EntityUtils.toString(response.getEntity()));
    }
}

It's weird because it should work as a container. Do you have any idea why is it requiring NTLM / Kerberos? I ask because I'm going to use it as part of a GitLab CI/CD.

Hi @BrunoVazCosta ,

The proxy server is enforcing the NTLM /Kerberos authentication. May be the proxy server is a product from Microsoft product. Can you check with your IT team?

Hi @rathnapandi, I have good news. I've commented the proxy settings in the env.properties and it worked! I've assumed I'd to set it but I don't have to. Even with 1.21.1 version. I'm sorry for the misunderstanding in how to use the CLI and many thanks for your support! You really did help.

Axway-API-Management-Plus / apim-cli