(Linux) Add Proxy configuration support

SwonVIP commented 3 years ago

Expected Behaviour

When using the aio-cli behind a corporate proxy the proxy should be somehow configurable or it should be able to use the environment variables

Actual Behaviour

aio-cli cannot connect to the internet for example to load the project list from AEMaaCS account.

Reproduce Scenario (including but not limited to)

Be behind a corporate proxy like proxy.example:8080 which supports NTLM authentication.

Steps to Reproduce

Launch aio and try to access your AEMaaCS information during the init steps.

Platform and Version

Linux Ubuntu (18.04) aio-cli:Latest node/npm: LTS

Sample Code that illustrates the problem

Cant priovide this only had temporary access to a AEMaaCS test account during AdaptTo() 2020

Logs taken while reproducing problem

Connection failed error message

meryllblanchet commented 3 years ago

Hi @SwonVIP and thanks for reaching out!

Could we ask you a bit more of details?

The output of the aio info command on your machine
Which was the use-case you were working on?

This is the first time we have a user facing proxy issues and would love to hear more.

Note: this should not be the reason for your issue, but fyi aio-cli currently supports nodeJS 10 and 12 LTS, as per our documentation.

cc @purplecabbage @sarahxxu

sarahxxu commented 3 years ago

@SwonVIP

our teammate @shazron found this -- https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-configure-proxy-on-ubuntu-18-04/ would you like to give it a try?

SwonVIP commented 3 years ago

Hi @meryllblanchet

Output of aio info:

aio info System: OS: Linux 5.4 Ubuntu 18.04.5 LTS (Bionic Beaver) CPU: (4) x64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz Memory: 23.24 GB / 31.12 GB Shell: 4.4.20 - /bin/bash Binaries: Node: 12.14.1 - ~/.nvm/versions/node/v12.14.1/bin/node Yarn: Not Found npm: 6.13.4 - ~/.nvm/versions/node/v12.14.1/bin/npm Virtualization: Docker: 19.03.13 - /usr/bin/docker npmGlobalPackages: @adobe/aio-cli: 3.7.0 CLI plugins: @adobe/aio-cli 3.7.0 (core) @adobe/aio-cli-plugin-app 3.5.2 (core) @adobe/aio-cli-plugin-auth 2.4.0 (core) @adobe/aio-cli-plugin-certificate 0.2.0 (core) @adobe/aio-cli-plugin-config 2.3.0 (core) @adobe/aio-cli-plugin-console 3.2.0 (core) @adobe/aio-cli-plugin-events 1.1.0 (core) @adobe/aio-cli-plugin-info 1.1.1 (core) @adobe/aio-cli-plugin-runtime 1.10.2 (core) @oclif/plugin-autocomplete 0.2.0 (core) @oclif/plugin-help 2.2.3 (core) @oclif/plugin-not-found 1.2.4 (core) @oclif/plugin-plugins 1.9.0 (core) @oclif/plugin-warn-if-update-available 1.7.0 (core)

So my use-case was to create or init an app/project for an adaptTo conference workshop I was participating:

aio app init adaptTo Retrieving information from Adobe I/O Console.. [CoreConsoleAPISDK:ERROR_GET_ORGANIZATIONS] FetchError: request to https://developers.adobe.io/console/organizations failed, reason: connect ETIMEDOUT 20.41.43.227:443 at ClientRequest. (/home/XXX/XXXXX/.nvm/versions/node/v12.14.1/lib/node_modules/@adobe/aio-cli/node_modules/swagger-client/node_modules/node-fetch/index.js:133:11) at ClientRequest.emit (events.js:223:5) at TLSSocket.socketErrorListener (_http_client.js:406:9) at TLSSocket.emit (events.js:223:5) at emitErrorNT (internal/streams/destroy.js:92:8) at emitErrorAndCloseNT (internal/streams/destroy.js:60:3) at processTicksAndRejections (internal/process/task_queues.js:81:21) { name: 'FetchError', message: 'request to https://developers.adobe.io/console/organizations failed, reason: connect ETIMEDOUT 20.41.43.227:443', type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }

You are about to initialize the project 'adaptTo'

@sarahxxu Unfortunately the global proxy environment variables are already set. However it is a proxy with NTLM authentication - maybe that could be the problem in this case.

I dont know if it is possible but it would be really nice if the aio would have an available flag to set or configure the proxy somehow. Maybe to even be more system independant - could be useful in certain situations.

shazron commented 3 years ago

For NTLM, we need to wrap all our service calls with an NTLM handshake. This will have to be a new feature @sarahxxu This is some sample code: https://gist.github.com/JoeStanton/64697ebcd865d6d1145d020475d7a0f4

sarahxxu commented 3 years ago

@shazron Thanks for the investigation. Let's put this as an improvement ticket for later! :)

meryllblanchet commented 3 years ago

@sarahxxu , I flagged this one as enhancement request accordingly.

aiojbot commented 3 years ago

JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-950

meryllblanchet commented 3 years ago

Hi @justinedelson , it seems that https://github.com/adobe/aio-cli-plugin-cloudmanager/issues/177 relates to this one. What about joining forces to handle the proxy support at CLI level?

cc @sarahxxu

justinedelson commented 3 years ago

@meryllblanchet I absolutely think that the mechanism to set proxy configuration should be consistent across plugins. I honestly don't know enough to say whether the mechanism to use that configuration can be shared easily. For example, aio-lib-ims (which the Cloud Manager plugin still doesn't use due to https://github.com/adobe/aio-lib-ims/issues/52, but that's a different topic) uses request-promise-native but aio-lib-cloudmanager uses cross-fetch.

So... I hate to say this, but I think what we need is aio-lib-fetch which would then be adopted by all of the libs (probably starting with aio-lib-ims).

meryllblanchet commented 3 years ago

@shazron any reason why we opted for request-promise-native in the context of the aio-lib-ims? AFAIK we usually leverage cross-fetch.

@justinedelson adobe/aio-lib-ims#52 hasn't been prioritized yet on our side. https://github.com/adobe/aio-lib-core-networking is the starting point you're thinking about.

cc @sarahxxu

shazron commented 3 years ago

@meryllblanchet I believe we can move off request-promise since it has been deprecated. Since this was implemented by another developer, there was no cohesiveness of library usage, which we can do now. We are using node-fetch in most of the libraries, and sometimes using cross-fetch (which uses node-fetch behind the scenes).

shazron commented 3 years ago

arobsco commented 2 years ago

Is proxy support still something that is being worked on?

pahupe commented 2 years ago

@arobsco : this has been implemented for aio. Not sure why this ticket is still open

arobsco commented 2 years ago

Thanks wasn't sure as having issues with the aio-cli app not connecting and only reason I can come up with is it's not honouring proxy settings but I'll do some more digging

pahupe commented 2 years ago

Please reach out to @shazron for implementation details, if needed

arobsco commented 2 years ago

@shazron it looks like this issue isn't fully resolved in that although https://github.com/adobe/aio-lib-core-networking is proxy aware and supports proxies, not all the plugins for AIO CLI use the networking package. For example https://github.com/adobe/aio-lib-console/blob/7ec79a21e83d8d4db6642b8ad6bb84a386a1fb14/src/index.js#L185-L190 uses Swagger API which is not configured to use networking package and therefore have proxy awareness

Changing the code to require the networking package and use custom fetch solves the issue at least for the aio-lib-console package

adobe / aio-cli