BUG - deployment to ABAP fails with 502 error from Axios

[tavnabisoft]

Related Feature

No related fuatures/issues found

Description

Deployment to ABAP using deploy-to-abap task fails with following error:

debug abap-deploy-task *** Retrieving application *** from /sap/opu/odata/UI5/ABAP_REPOSITORY_SRV, AxiosError: Request failed with status code 502

We're connecting to ABAP system through the gateway, deployment happens through fiori deploy --config ui5.yaml --yes command Full log of command is attached to an issue (all private parts are being masked with asterisk deploy-to-abap-logs.txt ).

Expected results

Application is deployed to ABAP system protected behind proxy

Actual results

Error being thrown

Version/Components/Environment

@sap/ux-ui5-tooling version: 1.14.1 Add any other context about the problem here OS:

• [ ] Mac OS
• [x] Windows
• [ ] Other

Root Cause Analysis

Problem

{describe the problem}

Fix

{describe the fix}

Why was it missed

{Some explanation why this issue might have been missed during normal development/testing cycle}

How can we avoid this

{if we don’t want to see this type of issues anymore what we should do to prevent}

[longieirl]

Hi @tavnabisoft

Can you please run the following commands;

curl -u 'username:password' $'https:/your-site-url/sap/bc/adt/discovery' -X GET -i -H 'X-Csrf-Token:Fetch' > output-tsk1.txt

curl -u 'username:password' $'https:/your-site-url/sap/opu/odata/UI5/ABAP_REPOSITORY_SRV/Repositories(\'ZIRN\')' -X GET -i -H 'X-Csrf-Token:Fetch' > output-tsk2.txt

Please update the username, password and site URL details to reflect your environment. I've appended the username:password properties as you have credentials appended as part of your output file. If this is not required, please remove -u 'username:password' from each of th respective commands.

This is a good test to validate, independent of our tooling if it's an issue with the ABAP target system and/or any local proxy/firewall as the HTTP error 502 indicates an upstream issue.

Can you also confirm any details related to your ABAP system? Any additional information about your system would also be appreciated, i.e., on-premise | cloud.

Thanks. John

[tavnabisoft]

Hello @longieirl ,

Thanks for quick response! I've attached output files for both commands. output-tsk1.txt output-tsk2.txt

ABAP system is on-premise. Here is some additional info that may be useful:
SAP_BASIS	752	0012	SAPK-75212INSAPBASIS	SAP Basis Component
SAP_ABA	752	0012	SAPK-75212INSAPABA	Cross-Application Component
SAP_GWFND	752	0012	SAPK-75212INSAPGWFND	SAP Gateway Foundation
SAP_UI	757	0003	SAPK-75703INSAPUI	User Interface Technology

There is a workaround for mentioned issue is by running in parallel ui5 serve command which refers to ui config with fiori-tools-proxy middleware which proxies requests to abap system. Here is the config of ui5.yaml file:

specVersion: "3.0"
metadata:
  name: ***
type: application

server:
  customMiddleware:
    # credentials are defined via environment variables vars: https://www.npmjs.com/package/@sap/ux-ui5-tooling#providing-credentials
    - name: fiori-tools-proxy
      afterMiddleware: compression
      configuration:
        ignoreCertError: false
        backend:
          - path: /sap/opu/odata
            url: https://***.com
          - path: /sap/public/bc
            url: https://***.com

Let me know if you need any further details.

Best Regards

[longieirl]

@tavnabisoft so you are running the command fiori deploy --config ui5.yaml --yes which is reading the ui5.yaml as the config. Can you share this config?

Also, when you ran the curl commands? Were they run with ui5-serve running in the background? If soo, can you please re-run with the ui5-server middleware stopped.

[tavnabisoft]

@longieirl we've run mentioned curl commands without running ui5 serve command in the background. So requests are executed with no issues without proxying by ui5 middleware.

Here is config which we're using for deployment to abap. If we're running deployment with ui5 serve in background, url of abap system is changed to http://localhost:8080:

specVersion: "3.0"
metadata:
  name: test-app
type: application

builder:
  resources:
    excludes:
      - /test/**
      - /localService/**
      - /index.html

  customTasks:

    - name: ui5-tooling-transpile-task
      afterTask: escapeNonAsciiCharacters
      configuration:
        debug: false
        removeConsoleStatements: true
        transformAsyncToPromise: true

    - name: deploy-to-abap
      afterTask: generateCachebusterInfo
      configuration:
        target:
          url: https://***.com
          auth: basic
        app:
          name: /TESTAPP
          description: "Test app"
          package: /TEST
          transport: ABC1324567
        credentials:
          username: env:SAP_GATEWAY_AUTH_USR
          password: env:SAP_GATEWAY_AUTH_PSW

[longieirl]

Morning @tavnabisoft, a couple of things to note, The curl commands are picking up the proxy Fortinet-Proxy so I'm wondering if the issue is related to this. The ui5 command must be picking up the proxy as well. When I look at the deploy-to-abap-logs.txt it shows the host as proxy.***.com, is this correct? Is that the expected proxy?

It might be worth reviewing the npm proxy settings to ensure the proxy is picked up; https://ga.support.sap.com/dtp/viewer/index.html#/tree/3046/actions/45995:48363:53593:53499

Also, when you run ui5 serve, it should indicate in the logs if a proxy is defined? Can you confirm this?

FYI, you can append the following exclude to the same level as app and target;

        exclude:
          - /test/

You also have testMode: true configured but this is not in your config. By default it should be false. I'm wondering where this is being picked up?

[tavnabisoft]

Hi @longieirl

The ui5 command must be picking up the proxy as well. When I look at the deploy-to-abap-logs.txt it shows the host as proxy.***.com, is this correct? Is that the expected proxy?

Yes, it is an expected proxy, the url has been masked. I've attached additional log for details, urls should be there: abap-deploy-logs-3.txt

It might be worth reviewing the npm proxy settings to ensure the proxy is picked up; https://ga.support.sap.com/dtp/viewer/index.html#/tree/3046/actions/45995:48363:53593:53499

Checked => proxy urls are set up

Also, when you run ui5 serve, it should indicate in the logs if a proxy is defined? Can you confirm this?

Yes, according to the logs proxy config is being picked. I've attached logs of ui5 serve command for reference: ui5-serve-logs.txt

FYI, you can append the following exclude to the same level as app and target;

Thanks for notice, we will adjust that :)

You also have testMode: true configured but this is not in your config. By default it should be false. I'm wondering where this is being picked up?

Yes, I've executed the task in test mode firstly (was defined in config of the deploy-to-abap task). The newly provided logs abap-deploy-logs-3.txt were executed with test mode being turned off

[longieirl]

Hi @tavnabisoft,

Thanks again for the logs; A couple of more questions!

1. Have you configured a SAP Fiori system using the URL and client as defined in your ui5 yaml configuration? For example, if you open VSCode and select SAP Fiori on the left navigation bar, is there is a system created matching the same host and client?
2. Can you please create a new scipt in your package.json or manipulate the existing script, for example;

   # Windows
   set DEBUG=* && fiori deploy --config ui5.yaml --yes
   # Linux/Mac
   DEBUG=* && fiori deploy --config ui5.yaml --yes

   You may need to tweak the fiori command to ensure it matches your deployment settings. Please attach the generated log, again, you may need to obfuscate any sensitive data.

This should hopefully output all the axios HTTP requests which might provide more insight to why the request is failing.

Thanks. John

[tavnabisoft]

Hello @longieirl,

1. System was added but test connection returns following error:

   2024-09-04 12:55:06.600 [error] The V2 request failed with error: Request failed with status code 502.
   2024-09-04 12:55:06.601 [error] The V4 request failed with error: Request failed with status code 502.

2. Please find attached logs of mentioned script: debug-all-deploy-log.txt

Thanks and regards

[longieirl]

Thank you again for the latest scripts.

I've a feeling the ui5 serve is working since that pulls in as many environment variables as it can find i.e.

        process.env.FIORI_TOOLS_PROXY ||
        process.env.npm_config_proxy ||
        process.env.npm_config_https_proxy ||
        process.env.http_proxy ||
        process.env.HTTP_PROXY ||
        process.env.https_proxy ||
        process.env.HTTPS_PROXY ||

But I'm wondering what the npm proxy settings are; if you run the following command, it will output the the variables used when npm is executed;

npm config list

Does this have a proxy setting? Can you please verify they are correct.

You can edit this file by running the following command, it will default to your system editor i.e. vi;

npm config edit

You can set them manually;

npm config set proxy http://<username><password>@proxy-server-url>:<port>
npm config set https-proxy http://<username><password>@proxy-server-url>:<port>

Once you have made any modifications, you can re-run the deploy command again.

Thanks.

[tavnabisoft]

Hello @longieirl,

Thanks for reply, find answers below:

Does this have a proxy setting? Can you please verify they are correct.

Proxy settings are set, here are the config values:

https-proxy = "http://user:***@proxy.domain.com:3128/"
noproxy = ["localhost,127.0.0.1"]
proxy = "http://user:***@proxy.domain.com:3128/"

Once you have made any modifications, you can re-run the deploy command again.

Mentioned modifications did not bring any change to deployment command results.

We've found that difference between fiori-tools-proxy middleware and deploy-to-abap task is that the first one uses https-proxy-agent module while the second one uses plain axios request. This is a known issue of making axios requests through the proxy, described here: https://stackoverflow.com/a/62277342/3307004.

1. fiori-tools-proxy middleware uses axios + https-proxy-agent: https://github.com/SAP/open-ux-tools/tree/main/packages/backend-proxy-middleware https://github.com/SAP/open-ux-tools/blob/main/packages/backend-proxy-middleware/src/base/proxy.ts#L1 https://github.com/SAP/open-ux-tools/blob/main/packages/backend-proxy-middleware/src/base/proxy.ts#L382
2. deploy-to-abap task uses axios.get directly without providing a proxy config: https://github.com/SAP/open-ux-tools/blob/main/packages/axios-extension/src/abap/ui5-abap-repository-service.ts#L132 (Class Ui5AbapRepositoryService extends OdataService) which is technically an Axios-Extension: https://github.com/SAP/open-ux-tools/blob/main/packages/axios-extension/src/base/odata-service.ts

Calling Axios directly without the https-proxy-agent leads to a wrong HTTP Request: Axios sends a GET Request while a valid proxy configuration exists: GET https://domain.com/sap/opu/odata/IWFND/CATALOGSERVICE;v=2/?sap-client=401 HTTP/1.1

This needs to be a CONNECT request to enable TLS: CONNECT domain.com:443 HTTP/1.1 https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT

Best Regards

[longieirl]

Ok! Thank you for the investigation, its appreciated. Let me do some more analysis and I will hopefully come up with a plan of action to get this resolved.

[longieirl]

Soo I've been playing with the following docker setup;

---
services:
  proxy:
    image: ubuntu/squid:latest
    container_name: squidproxy
    restart: unless-stopped
    ports:
      - 3128:3128

Then setting the npm proxy using;

npm config set proxy http://localhost:3128
npm config set https-proxy http://localhost:3128

And I'm able to deploy as expected, in the squid proxy logs, I can see the GET and POST requests.

--- Update

I've tweaked the docker compose file as follows;

version: "3.8"

services:
  proxy:
    build: 
      context: ./
    container_name: squidproxy
    restart: unless-stopped
    environment:
      TZ: Europe/Dublin
    ports:
      - 3128:3128

Created a new Dockerfile;

FROM ubuntu/squid

COPY passwords /etc/squid/passwords
COPY squid.conf /etc/squid/squid.conf

RUN chown proxy:proxy /etc/squid/squid.conf /etc/squid/passwords

Created a new password file using foo and boo and using the following npm settings as follows;

npm config set proxy http://foo:boo@localhost:3128
npm config set https-proxy http://foo:boo@localhost:3128

and I'm able to connect correctly to the backend. Again, if I put in the incorrect user/pass, it will block the connection to the ABAP target system.

[tavnabisoft]

Hello @longieirl , Thanks for investigation, but I believe that difference in original case and your scenario is that request should go to https target url over the http proxy. Axios does not support this scenario out the box and require usage of https-proxy-agent library to execute such requests. You can see details in Stackoverflow questiion and Github Issue of axios.

[longieirl]

Morning,

I've managed to hack a squid proxy with SSL certs and I'm getting a 503 when using axios. The same request using a curl with the proxy flag, is working as expected.

info abap-deploy-task ZTESTJL Creating archive with UI5 build result.
info abap-deploy-task ZTESTJL Archive created.
info abap-deploy-task ZTESTJL Starting to deploy.
error abap-deploy-task ZTESTJL Deployment has failed.
error abap-deploy-task ZTESTJL Change logging level to debug your issue
error abap-deploy-task ZTESTJL  (see examples https://github.com/SAP/open-ux-tools/tree/main/packages/deploy-tooling#configuration-with-logging-enabled)
error builder:custom deploy-to-abap Request failed with status code 503
Command deploy failed with error : Request failed with status code 503

I will try to address this issue over the coming days. Thanks again.

John

[tavnabisoft]

Hello @longieirl ,

Do you have updates on the topic?

[zinserjan]

Hello @longieirl , Thanks for investigation, but I believe that difference in original case and your scenario is that request should go to https target url over the http proxy. Axios does not support this scenario out the box and require usage of https-proxy-agent library to execute such requests. You can see details in Stackoverflow questiion and Github Issue of axios.

I noticed the same behavior with the abap deploy task some time ago. All other middleware are working fine, but they don't use @sap-ux/axios-extension and use a different http lib.

The following didn't work for me

1. HTTP over HTTP proxy doesn't work (I wasn't able to reach the system - I tested this only a few minutes)
2. HTTPs over HTTP proxy doesn't work
3. ignoreCertErrors was ignored

I was able to fix 2 and 3 with the following patch and just replaced the whole @sap-ux/axios-extension module in my project via npm@overrides. I assume it will only work for HTTPs requests, because httpAgent isn't patched for HTTP yet.

Note: I also had to patch https-proxy-agent dirty because rejectUnauthorized was ignored (see TooTallNate/proxy-agents/issues#89)

diff --git a/packages/axios-extension/package.json b/packages/axios-extension/package.json
index 71411a758..db43a0e2f 100644
--- a/packages/axios-extension/package.json
+++ b/packages/axios-extension/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@sap-ux/axios-extension",
-    "version": "1.16.2",
+    "version": "1.16.2-patched",
     "description": "Extension of the Axios module adding convenience methods to interact with SAP systems especially with OData services.",
     "repository": {
         "type": "git",
@@ -31,8 +31,10 @@
         "axios": "1.6.8",
         "detect-content-type": "1.2.0",
         "fast-xml-parser": "4.4.1",
+        "https-proxy-agent": "7.0.5",
         "lodash": "4.17.21",
         "open": "7.0.3",
+        "proxy-from-env": "1.1.0",
         "qs": "6.11.0",
         "xpath": "0.0.33",
         "@xmldom/xmldom": "0.8.10"
diff --git a/packages/axios-extension/src/factory.ts b/packages/axios-extension/src/factory.ts
index 886fe8aa0..ee8d5edbd 100644
--- a/packages/axios-extension/src/factory.ts
+++ b/packages/axios-extension/src/factory.ts
@@ -1,4 +1,8 @@
+/* eslint-disable jsdoc/require-jsdoc */
 import type { AxiosRequestConfig, InternalAxiosRequestConfig } from 'axios';
+import type { HttpsProxyAgentOptions } from 'https-proxy-agent';
+import { HttpsProxyAgent } from 'https-proxy-agent';
+import { getProxyForUrl } from 'proxy-from-env';
 import cloneDeep from 'lodash/cloneDeep';
 import type { Destination } from '@sap-ux/btp-utils';
 import {
@@ -22,6 +26,19 @@ import { AbapServiceProvider } from './abap';
 import { inspect } from 'util';
 import { TlsPatch } from './base/patchTls';

+// eslint-disable-next-line jsdoc/require-jsdoc
+export class PatchedHttpsProxyAgent<Uri extends string> extends HttpsProxyAgent<Uri> {
+    private readonly extraOptions: any;
+
+    constructor(proxy: Uri | URL, opts?: HttpsProxyAgentOptions<Uri>) {
+        super(proxy, opts);
+        this.extraOptions = opts;
+    }
+
+    async connect(req: any, opts: any) {
+        return super.connect(req, Object.assign({}, this.extraOptions, opts));
+    }
+}
 type Class<T> = new (...args: any[]) => T;

 /**
@@ -36,9 +53,24 @@ function createInstance<T extends ServiceProvider>(
     config: AxiosRequestConfig & Partial<ProviderConfiguration>
 ): T {
     const providerConfig: AxiosRequestConfig & Partial<ProviderConfiguration> = cloneDeep(config);
-    providerConfig.httpsAgent = new HttpsAgent({
-        rejectUnauthorized: !providerConfig.ignoreCertErrors
-    });
+
+    if (config.baseURL) {
+        const proxyUrl = getProxyForUrl(config.baseURL);
+        if (proxyUrl) {
+            // axios doesn't handle proxies correctly
+            providerConfig.httpsAgent = new PatchedHttpsProxyAgent(proxyUrl, {
+                rejectUnauthorized: !providerConfig.ignoreCertErrors
+            });
+            providerConfig.proxy = false; // disable proxy handling of axios
+        }
+    }
+
+    if (!providerConfig.httpsAgent) {
+        providerConfig.httpsAgent = new HttpsAgent({
+            rejectUnauthorized: !providerConfig.ignoreCertErrors
+        });
+    }
+
     delete providerConfig.ignoreCertErrors;
     providerConfig.withCredentials = providerConfig?.auth && Object.keys(providerConfig.auth).length > 0;

[tavnabisoft]

Hello @zinserjan , Thanks for proposed solution, however the overriding is not working for us as we're using @sap/ux-ui5-tooling package and it contains all dependencies (including axios-extension) bundled & minimized into single js file: @sap\ux-ui5-tooling\dist\cli\index.js. Replacing just axios-extension module in this file is not easy as all naming/minimization is bound with all other sub-modules of ux-tooling. Also, open-ux-tools does not provide command to bundle all dependencies into single source. What package did you use for applying the workaround? @longieirl Is there a way to create single bundle file with all sub-modules, so it can substitute @sap\ux-ui5-tooling\dist\cli\index.js in node modules?

[zinserjan]

@tavnabisoft Thankfully, I don't use @sap/ux-ui5-tooling in this project. Generators didn't work in this case and that's why I just did the setup with the new ui5 cli v4 on my own. Nearly every function of @sap/ux-ui5-tooling can be replaced with a package of this repo. And my setup is anyway pretty custom due to odata2ts and prettier:

  "dependencies": {
    "@odata2ts/converter-v2-to-v4": "^0.4.0",
    "@odata2ts/http-client-jquery": "^0.8.2",
    "@odata2ts/odata-service": "^0.19.3"
  },
  "devDependencies": {
    "@odata2ts/odata2ts": "^0.36.2",
    "@prettier/plugin-xml": "^3.4.1",
    "@sap-ux/backend-proxy-middleware": "^0.8.4",
    "@sap-ux/deploy-tooling": "^0.15.7",
    "@sap-ux/eslint-plugin-fiori-tools": "^0.5.0",
    "@sap-ux/preview-middleware": "^0.16.35",
    "@sap-ux/reload-middleware": "^0.2.0",
    "@sap-ux/ui5-middleware-fe-mockserver": "^2.2.71",
    "@sap-ux/ui5-proxy-middleware": "^1.4.6",
    "@sapui5/types": "~1.120.14",
    "@ui5/builder": "^4.0.1",
    "@ui5/cli": "^4.0.3",
    "eslint": "^8.57.0",
    "prettier": "^3.3.3",
    "prettier-plugin-imports": "^4.2.11",
    "prettier-plugin-packagejson": "^2.5.1",
    "prettier-plugin-properties": "^0.3.0",
    "typescript": "^5.1.6",
    "ui5-tooling-modules": "^3.9.1",
    "ui5-tooling-transpile": "^3.4.7"
  },
  "overrides": {
    "@sap-ux/axios-extension": "file:.packages/sap-ux-axios-extension-1.16.2-patched.tgz"
  }