search

googleapis / nodejs-firestore

Node.js client for Google Cloud Firestore: a NoSQL document database built for automatic scaling, high performance, and ease of application development.
https://cloud.google.com/firestore/
Apache License 2.0
640 stars 149 forks source link

Firestore Client Does Not Work Behind a Proxy #493

Closed hiranya911 closed 4 years ago

hiranya911 commented 5 years ago

Environment details

Steps to reproduce

Trying to access Firestore through a simple HTTP proxy. According to the instructions in this article, Node.js GRPC supports specifying a proxy via an environment variable.

  1. Set the HTTPS_PROXY environment variable
  2. Try to access any Firestore API

GRPC debug log

D1215 12:10:04.861744405   17298 dns_resolver.cc:338]        Using native dns resolver
D1215 12:10:04.937711172   17298 dns_resolver.cc:279]        Start resolving.
I1215 12:10:04.964020153   17298 ssl_transport_security.cc:174] OpenSSL callback has already been set.
I1215 12:10:05.012894812   17298 handshaker.cc:141]          handshake_manager 0x34c5280: adding handshaker http_connect [0x34c53d0] at index 0
I1215 12:10:05.013020674   17298 handshaker.cc:141]          handshake_manager 0x34c5280: adding handshaker security [0x34df1a0] at index 1
I1215 12:10:05.013071119   17298 handshaker.cc:212]          handshake_manager 0x34c5280: error="No Error" shutdown=0 index=0, args={endpoint=0x3341fb0, args=0x34c2c00 {size=9: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7fe7bf9e6520, grpc.channel_credentials=0x2d3cfc0, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7fe7bf772e50, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x2e8f690, grpc.subchannel_address=ipv6:[2607:f8b0:4005:805::200a]:443}, read_buffer=0x34df630 (length=0), exit_early=0}
I1215 12:10:05.013076542   17298 handshaker.cc:258]          handshake_manager 0x34c5280: calling handshaker http_connect [0x34c53d0] at index 0
I1215 12:10:05.013104710   17298 handshaker.cc:212]          handshake_manager 0x34c5280: error="No Error" shutdown=0 index=1, args={endpoint=0x3341fb0, args=0x34c2c00 {size=9: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7fe7bf9e6520, grpc.channel_credentials=0x2d3cfc0, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7fe7bf772e50, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x2e8f690, grpc.subchannel_address=ipv6:[2607:f8b0:4005:805::200a]:443}, read_buffer=0x34df630 (length=0), exit_early=0}
I1215 12:10:05.013122409   17298 handshaker.cc:258]          handshake_manager 0x34c5280: calling handshaker security [0x34df1a0] at index 1
I1215 12:10:05.070240703   17298 handshaker.cc:212]          handshake_manager 0x34c5280: error="No Error" shutdown=0 index=2, args={endpoint=0x34e02c0, args=0x34e7230 {size=10: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7fe7bf9e6520, grpc.channel_credentials=0x2d3cfc0, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7fe7bf772e50, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x2e8f690, grpc.subchannel_address=ipv6:[2607:f8b0:4005:805::200a]:443, grpc.auth_context=0x34e9150}, read_buffer=0x34df630 (length=0), exit_early=0}
I1215 12:10:05.070254897   17298 handshaker.cc:245]          handshake_manager 0x34c5280: handshaking complete -- scheduling on_handshake_done with error="No Error"
I1215 12:10:05.070331159   17298 subchannel.cc:656]          New connected subchannel at 0x34d4a60 for subchannel 0x2f7a550
Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

Auth error:Error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

(node:17298) UnhandledPromiseRejectionWarning: Error: 14 UNAVAILABLE: Getting metadata from plugin failed with error: write EPROTO 140633384175424:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:

    at Object.exports.createStatusError (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/common.js:87:15)
    at ClientReadableStream._emitStatusIfDone (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client.js:235:26)
    at ClientReadableStream._receiveStatus (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client.js:213:8)
    at Object.onReceiveStatus (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client_interceptors.js:1256:15)
    at InterceptingListener._callNext (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client_interceptors.js:564:42)
    at InterceptingListener.onReceiveStatus (/home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client_interceptors.js:614:8)
    at /home/hiranya/Projects/firebase/proxy-config/node-example/node_modules/grpc/src/client_interceptors.js:1019:24

HTTP proxy log (Squid proxy access log)

1544904335.228      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904337.509      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904342.295      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904342.403      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904605.087      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904607.963      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904612.874      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904612.984      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904615.098      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904619.662      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904619.768      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904622.422      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904626.594      0 172.17.0.1 TAG_NONE/400 4038  1%84k%12%C2=%14%BD%1A6%D8%CC%C4%9ELz%DA%60%B6%I%8E. - HIER_NONE/- text/html
1544904626.698      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904629.113      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904634.095      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904634.199      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904636.265      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html
1544904640.374      0 172.17.0.1 TAG_NONE/400 4390 NONE error:invalid-request - HIER_NONE/- text/html

Note that this problem is unique to the Firestore Node.js client. I've tried the same with Java and Go clients where this use case works fine.

hiranya911 commented 5 years ago

It seems the auth library is at fault here. It's not connecting to the HTTP proxy correctly.

If I just set the HTTP_PROXY variable, I can get Firestore to go through the proxy. But in that case the credentials will not go through the proxy.

D1215 13:57:28.712727343   26718 dns_resolver.cc:338]        Using native dns resolver
D1215 13:57:28.787131284   26718 dns_resolver.cc:279]        Start resolving.
I1215 13:57:28.800943867   26718 ssl_transport_security.cc:174] OpenSSL callback has already been set.
I1215 13:57:28.821523393   26718 handshaker.cc:141]          handshake_manager 0x35d2170: adding handshaker http_connect [0x338d2c0] at index 0
I1215 13:57:28.821620747   26718 handshaker.cc:141]          handshake_manager 0x35d2170: adding handshaker security [0x35d1090] at index 1
I1215 13:57:28.821647440   26718 handshaker.cc:212]          handshake_manager 0x35d2170: error="No Error" shutdown=0 index=0, args={endpoint=0x33cfed0, args=0x3318ca0 {size=10: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7f406739f520, grpc.channel_credentials=0x33b2480, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7f406712be50, grpc.http_connect_server=firestore.googleapis.com:443, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x34ca5e0, grpc.subchannel_address=ipv4:127.0.0.1:3128}, read_buffer=0x338e500 (length=0), exit_early=0}
I1215 13:57:28.821656912   26718 handshaker.cc:258]          handshake_manager 0x35d2170: calling handshaker http_connect [0x338d2c0] at index 0
I1215 13:57:28.821662952   26718 http_connect_handshaker.cc:300] Connecting to server firestore.googleapis.com:443 via HTTP proxy ipv4:127.0.0.1:3128
I1215 13:57:28.851203756   26718 handshaker.cc:212]          handshake_manager 0x35d2170: error="No Error" shutdown=0 index=1, args={endpoint=0x33cfed0, args=0x3318ca0 {size=10: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7f406739f520, grpc.channel_credentials=0x33b2480, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7f406712be50, grpc.http_connect_server=firestore.googleapis.com:443, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x34ca5e0, grpc.subchannel_address=ipv4:127.0.0.1:3128}, read_buffer=0x338e500 (length=0), exit_early=0}
I1215 13:57:28.851230553   26718 handshaker.cc:258]          handshake_manager 0x35d2170: calling handshaker security [0x35d1090] at index 1
I1215 13:57:28.901747144   26718 handshaker.cc:212]          handshake_manager 0x35d2170: error="No Error" shutdown=0 index=2, args={endpoint=0x33b3220, args=0x33acc00 {size=11: grpc.primary_user_agent=grpc-node/1.16.1, grpc.client_channel_factory=0x7f406739f520, grpc.channel_credentials=0x33b2480, grpc.server_uri=dns:///firestore.googleapis.com:443, grpc.channelz_channel_node_creation_func=0x7f406712be50, grpc.http_connect_server=firestore.googleapis.com:443, grpc.default_authority=firestore.googleapis.com:443, grpc.http2_scheme=https, grpc.security_connector=0x34ca5e0, grpc.subchannel_address=ipv4:127.0.0.1:3128, grpc.auth_context=0x33c1b20}, read_buffer=0x338e500 (length=0), exit_early=0}
I1215 13:57:28.901762578   26718 handshaker.cc:245]          handshake_manager 0x35d2170: handshaking complete -- scheduling on_handshake_done with error="No Error"
I1215 13:57:28.901858190   26718 subchannel.cc:656]          New connected subchannel at 0x3529ab0 for subchannel 0x35c0250
test-topic

Squid log:

1544911049.455    635 172.17.0.1 TCP_TUNNEL/200 4455 CONNECT firestore.googleapis.com:443 - HIER_DIRECT/216.58.195.74 -

It seems I must set the HTTPS_PROXY variable for the credentials to go through the proxy, but it doesn't work with plain HTTP.

jkwlui commented 5 years ago

@hiranya911 We've release 0.21.0 which addressed the issue with running behind a proxy. Can you try now and see if it resolves it? Thanks!

jkwlui commented 5 years ago

@stephenplusplus I understand you have a test utility for proxy issues - can you please take a look at this? Thanks :)

JustinBeckwith commented 5 years ago

@stephenplusplus will help with verifying this

stephenplusplus commented 5 years ago

I believe @ajaaym found a solution:

can you please try setting both env variable https_proxy & HTTPS_PROXY GRPC uses https_proxy and authentication which goes through normal node request uses HTTPS_PROXY.

amammay commented 5 years ago

@JustinBeckwith @jkwlui with the release of 2.1.1 i can no longer hit firestore over a proxy. i have tried setting https_proxy, HTTPS_PROXY, http_proxy, HTTP_PROXY. It looks like 2.1.1 added "@grpc/grpc-js": "0.4.0". and from taking a quick look in that repo i dont see anything about grpc-js accepting proxy support? What do either of you think would be the best way to move forward?

stephenplusplus commented 5 years ago

That's unfortunate, I will look into that. I believe you could use a workaround where you provide the proxy endpoint programmatically in the constructor. @schmidt-sebastian is that possible? In other languages, it would look something like:

new Client({ apiEndpoint: 'https://localhost:8080' })
new Client({ host: 'localhost', port: '8080' })
schmidt-sebastian commented 5 years ago

The recommended syntax is:

new Client({ host: 'localhost:8080' })

apiEndpoint and servicePath should also work.

Note: We now also support non-SSL endpoints directly:

new Client({ host: 'localhost:8080'; ssl: false });
amammay commented 5 years ago

so that would work fine, but im behind a corporate proxy and need to pass in proxy authentication credentials. whenever i do so i get a Value for argument "settings.host" is not a valid host.

@schmidt-sebastian @stephenplusplus

in addition im getting this issue across other google apis projects like dialogflow or probably anything that is using grpc-js.

schmidt-sebastian commented 5 years ago

@amammay How do you pass these credentials? If they are HTTP headers, you can pass them in a such

const firestore = new Firestore({
  customHeaders: {
    "X-Name": "Value"
  }
});
amammay commented 5 years ago

@schmidt-sebastian well passing in the credentials would be of the format of

http://[user]:[pass]@hostname:port

and for the firebase settings you can only pass in host and port as https://github.com/googleapis/nodejs-firestore/blob/master/types/firestore.d.ts#L56

schmidt-sebastian commented 5 years ago

FWIW, we also support servicePath and apiEndpoint, which are passed directly to the underlying layer. I haven't yet been able to confirm that the support passing credentials as you suggested.

schmidt-sebastian commented 4 years ago

@hiranya911 Is this still an issue? We are now using Node's HTTP2 module for our networking.

hiranya911 commented 4 years ago

I've never tried this scenario with a proxy that requires authentication. Does the new implementation still support the https_proxy environment variable?

fatherOfLegends commented 4 years ago

I have not been able to successfully proxy any Firestore calls. I can proxy the token call but no requests to Firestore actually hit the proxy. I am using the latest version.

thebrianchen commented 4 years ago

The admin SDK uses grpc-js, which currently does not support proxies (issue). However, the native grpc library does support proxies using HTTP_PROXY.

One way to circumvent this limitation would be to provide a the native GRPC implementation to the admin SDK via the constructor argument as detailed in this comment, which also details why we use grpc-js instead of native grpc.

fatherOfLegends commented 4 years ago

@thebrianchen thanks for the tip. I gave this a shot, configured my app as the comment suggests, and using Charles proxy on my machine and I see the auth call hit the proxy but I do not see any calls to my Firestore database that hit the proxy. "firebase-admin": "8.8.0" "grpc": "1.24.2"

fatherOfLegends commented 4 years ago

@thebrianchen my apologies, I am hitting the proxy now I was using the wrong casing for the environment variable. HTTP_PROXY !== http_proxy

fatherOfLegends commented 4 years ago

@thebrianchen thank you!!

stephenplusplus commented 4 years ago

@zamnuts would you mind giving this a glance when you have a chance? From your recent experience with proxy support inside grpc-node, maybe we've resolved this already? And if not, do you know what issues are standing in the way?

schmidt-sebastian commented 4 years ago

@grpc/grpc-js 0.7.x adds supports for the HTTP_PROXY environment variable. We need to update our dependency and then we should be able to close this issue.

schmidt-sebastian commented 4 years ago

If you update your dependencies, you should now be using @grpc/grpc-js@0.7.5.