search

googleapis / nodejs-secret-manager

This repository is deprecated. All of its content and history has been moved to googleapis/google-cloud-node.
https://cloud.google.com/secret-manager
Apache License 2.0
87 stars 31 forks source link

Client fails silently when using https_proxy, http_proxy, HTTPS_PROXY, HTTP_PROXY environment variables and proxy server #92

Closed kuccello closed 4 years ago

kuccello commented 4 years ago

Environment details

Steps to reproduce

  1. Setup a proxy server (eg. Squid) with no filtering - allow all pass through
  2. Setup a GCP project with Secrets Manager - add a key/value to your GSM project
  3. Run this code: https://gist.github.com/kuccello/35996a41441422a7239960393fcd9045

You will observe that when using the nodejs-secret-manager (SecretManagerServiceClient) it fails silently when the environment defines a proxy using variables https_proxy, http_proxy, HTTPS_PROXY, HTTP_PROXY. This is unexpected - if it fails it should throw an exception, produce an error message or something.

bcoe commented 4 years ago

@kuccello double check that you're on @grpc/grpc-js@0.7.7, as proxy support was added fairly recently. The variables that should be supported are http_proxy, https_proxy, and grpc_proxy.

Assuming that you are still bumping into errors, and it's simply failing silently, this seems like a bug we should also raise on the @grpc/grpc-js repo. CC: @murgatroid99.

alexander-fenster commented 4 years ago

@kuccello also, just to double check that it's @grpc/grpc-js problem and not a proxy issue, can you try this:

npm install grpc

and then in your code

const grpc = require('grpc')
const client = new SecretManagerServiceClient({grpc})

and let us know if this one works or not? Thank you!

murgatroid99 commented 4 years ago

I have published grpc-js version 0.7.8 with a change that might fix the bug and should at least make it fail loudly. If you update your dependencies so that you have that version you should see a difference.

kuccello commented 4 years ago

To clarify - I ran a set of different env combinations and have discovered that it is only the lowercase http_proxy and https_proxy variables that are resulting in the silent error. I will update my npm libs and try your possible fix @murgatroid99 and report back.

kuccello commented 4 years ago

@alexander-fenster - I did as you suggested (installed grpc directly - see below) and the observed behavior is that the process now hangs with no error still. I had to force kill the container (it would not respond to crtl-c) - this is new behavior. I don't believe this is a down stream lib issue.

@bcoe - here is my deps list from the test project I setup to show this issue: Note that I was testing firestore and algolia access via squid separately

"dependencies": {
    "@google-cloud/secret-manager": "^2.1.0",
    "algoliasearch": "^4.1.0",
    "firestore-admin": "^1.3.0",
    "grpc": "^1.24.2",
    "http-proxy-agent": "^4.0.1",
    "https-proxy-agent": "^5.0.0"
  }

In the node modules directory I am seeing what looks like two grpc libs:

Additional Context: I am running this squid proxy docker container locally - built from source https://github.com/sameersbn/docker-squid and my proxy config is as follows.

Note: I do know that Squid 3.5.x does not really support HTTP/2 - the reason I filed this issue is that there seems to be missing error reporting from the lib.

Proxy Conf

#Recommended minimum configuration:
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 0.0.0.0/8 192.168.100.0/24 192.168.101.0/24
acl SSL_ports port 443
acl Safe_ports port 80      # http
acl Safe_ports port 21      # ftp
acl Safe_ports port 443     # https
acl Safe_ports port 70      # gopher
acl Safe_ports port 210     # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280     # http-mgmt
acl Safe_ports port 488     # gss-http
acl Safe_ports port 591     # filemaker
acl Safe_ports port 777     # multiling http

acl CONNECT method CONNECT

# http_access allow manager localhost
# http_access deny manager
# http_access deny !Safe_ports

# http_access deny to_localhost
# icp_access deny all
# htcp_access deny all

http_port 3128
hierarchy_stoplist cgi-bin ?
# access_log /var/log/squid3/access.log squid

#Suggested default:
refresh_pattern ^ftp:       1440    20% 10080
refresh_pattern ^gopher:    1440    0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern .       0   20% 4320
# Leave coredumps in the first cache dir
coredump_dir /squid/var/cache/squid

# Allow all machines to all sites
http_access allow all

Dockerfile

FROM node:12.16.1-alpine

WORKDIR /app/gsmtest

COPY . .

RUN npm i
RUN npm rebuild

CMD ["node","src/gsm-proxy.js"]
docker run -it \
  --env GOOGLE_APPLICATION_CREDENTIALS="/app/gsmtest/creds/constellation-gsm-key.json" \
  --env http_proxy="http://127.0.0.1:31280" \
  --env http_proxy="http://127.0.0.1:31280" \
  --name test-lower-proxy-gsm \
 gsm-test:latest
About to call GSM...
/app/gsmtest/creds/constellation-gsm-key.json
buildSecretFetcher called for  cp_algolia latest
[AsyncFunction]
getSecret called for  projects/REDACTED-PROJECT-ID/secrets/cp_algolia cp_algolia latest
Script end.
kuccello commented 4 years ago

A few other hints:

When I run without proxy the code executes as expected

docker run -it --env  GOOGLE_APPLICATION_CREDENTIALS="/app/gsmtest/creds/constellation-gsm-key.json"  gsm-test:latest

Output:

$>docker run -it --env  GOOGLE_APPLICATION_CREDENTIALS="/app/gsmtest/creds/constellation-gsm-key.json"  gsm-test:latest
About to call GSM...
/app/gsmtest/creds/constellation-gsm-key.json
buildSecretFetcher called for  REDACTED latest
[AsyncFunction]
getSecret called for  projects/REDACTED-PROJECT-ID/secrets/REDACTED-KEY-NAME latest
Script end.
[
  {
    name: 'projects/REDACTED-PROJECT-ID/secrets/REDACTED-KEY-NAME/versions/1',
    payload: {
      data: <Buffer 7b 0a 20 20 22 61 70 70 49 64 22 3a 20 22 42 35 35 52 55 4c 4c 57 53 44 22 2c 0a 20 20 22 61 64 6d 69 6e 4b 65 79 22 3a 20 22 39 31 33 62 64 37 62 36 ... 27 more bytes>
    }
  },
  null,
  null
]
Response received [
  {
    name: 'projects/REDACTED-PROJECT-ID/secrets/REDACTED-KEY-NAME/versions/1',
    payload: {
      data: <Buffer 7b 0a 20 20 22 61 70 70 49 64 22 3a 20 22 42 35 35 52 55 4c 4c 57 53 44 22 2c 0a 20 20 22 61 64 6d 69 6e 4b 65 79 22 3a 20 22 39 31 33 62 64 37 62 36 ... 27 more bytes>
    }
  },
  null,
  null
]

But when run with upper case proxy vars: HTTP_PROXY, HTTPS_PROXY

docker run -it \
  --env GOOGLE_APPLICATION_CREDENTIALS="/app/gsmtest/creds/constellation-gsm-key.json" \
  --env HTTP_PROXY="http://127.0.0.1:31280" \
  --env HTTPS_PROXY="http://127.0.0.1:31280" \
  --name test-upper-proxy-gsm \
 gsm-test:latest

Result

About to call GSM...
/app/gsmtest/creds/constellation-gsm-key.json
buildSecretFetcher called for  REDACTED latest
[AsyncFunction]
getSecret called for  projects/REDACTED/secrets/REDACTED REDACTED latest
Script end.
Error: ECONNREFUSED undefined: Getting metadata from plugin failed with error: request to https://www.googleapis.com/oauth2/v4/token failed, reason: connect ECONNREFUSED 127.0.0.1:31280
    at Object.callErrorFromStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client.js:174:52)
    at Object.onReceiveStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:340:141)
    at Object.onReceiveStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:303:181)
    at Http2CallStream.outputStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call-stream.js:114:27)
    at Http2CallStream.maybeOutputStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call-stream.js:153:22)
    at Http2CallStream.endCall (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call-stream.js:140:18)
    at Http2CallStream.cancelWithStatus (/app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call-stream.js:441:14)
    at /app/gsmtest/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/channel.js:202:36
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  code: 'ECONNREFUSED',
  details: 'Getting metadata from plugin failed with error: request to https://www.googleapis.com/oauth2/v4/token failed, reason: connect ECONNREFUSED 127.0.0.1:31280',
  metadata: Metadata { internalRepr: Map {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}
Response received null

When run with lowercase proxy env vars: http_proxy, https_proxy

Fails silently

docker run -it \
  --env GOOGLE_APPLICATION_CREDENTIALS="/app/gsmtest/creds/constellation-gsm-key.json" \
  --env http_proxy="http://127.0.0.1:31280" \
  --env http_proxy="http://127.0.0.1:31280" \
  --name test-lower-proxy-gsm \
 gsm-test:latest

Result

About to call GSM...
/app/gsmtest/creds/constellation-gsm-key.json
buildSecretFetcher called for  REDACTED latest
[AsyncFunction]
getSecret called for  projects/REDACTED/secrets/REDACTED REDACTED latest
Script end.

Currently with the additional setup suggested by @alexander-fenster the process hangs - prior to adding the grpc into the client config during creation the process exited silently and without error or data.

kuccello commented 4 years ago

@murgatroid99 - I installed the latest grpc-js - 0.7.8 and no change in behavior

"dependencies": {
    "@google-cloud/secret-manager": "^2.1.0",
    "@grpc/grpc-js": "^0.7.8",
    "algoliasearch": "^4.1.0",
    "firestore-admin": "^1.3.0",
    "grpc": "^1.24.2",
    "http-proxy-agent": "^4.0.1",
    "https-proxy-agent": "^5.0.0"
  }
murgatroid99 commented 4 years ago

Can you add in the environment variables GRPC_TRACE=all and GRPC_VERBOSITY=DEBUG (with the lower case http_proxy environment variables) and share the output?

kuccello commented 4 years ago

@murgatroid99 - could you email me and I will do a drive share of the trace - cursory pass shows there may be account specific info in the log and I would prefer not to make that public. My email is my github username at gmail.

kuccello commented 4 years ago

Shared the log with you @murgatroid99

murgatroid99 commented 4 years ago

I published some new proxy handling code in @grpc/grpc-js version 0.8.1. I believe you can try it out by adding an explicit dependency on @grpc/grpc-js and loading it in, similar to @alexander-fenster's suggestion earlier for trying out grpc: install like this:

npm install @grpc/grpc-js@^0.8.1

and then in your code:

const grpc = require('@grpc/grpc-js')
const client = new SecretManagerServiceClient({grpc})
murgatroid99 commented 4 years ago

An update on my last comment: I have now published grpc-js version 1.0, and it includes a fix that at least one person has used to traverse a proxy.

bcoe commented 4 years ago

@kuccello I think this should be (hopefully) addressed by the latest release of @grpc/grpc-js, let us know if it isn't 👍