aws-amplify / amplify-cli

The AWS Amplify CLI is a toolchain for simplifying serverless web and mobile development.
Apache License 2.0
2.81k stars 819 forks source link

Local function debug fails with timeout when calling appsync endpoint #2450

Closed diego-palmeira closed 5 years ago

diego-palmeira commented 5 years ago

Hi, Context: I'm working in a project which includes a react ui part with an api graphql, and another part in the same project with a lambda function. I want to reuse the api graphql to call the dynamodb using the same queries, etc. In order to use AWS AppSync in lambda functions, after try a lot of ways and appsync client libraries, I'm finally following this great tutorial at:
https://janhesters.com/how-to-use-aws-appsync-in-lambda-functions/

Problem: I have configured my local VSC to debug locally my lambda function and it launch a command: amplify function invoke myfunction

It runs ok, but when I hit my API using Postman, it reach a line in my code which calls the remote appsync endpoint:

data = await client.query({ query });

and hangs as follows: Promise { } and after some minutes it fails with a TIMEOUT: 'request to https://xxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 52.85.219.177:443'

On the other hand, I'm using cntlm local proxy on localhost 3128, with Ubuntu 14.04, and behind a corporative proxy. I have no problems with amplify init, or any other tool, in fact all my local tools points to cntlm to avoid configurate the company proxy on each one.

I have my proxy variables well configured, and I added also them to the launch.json as follows just in case, but with the same result:

{ "version": "0.2.0", "configurations": [ { "type": "node", "request": "launch", "name": "Launch myfunction", "program": "/usr/local/bin/amplify", "args": [ "function", "invoke", "myfunction" ], "env": { "ENV": "dev", "REGION": "xxxx", "AUTH_SOSAUTH_USERPOOLID": "xxxx", "ANALYTICS_SOSANALYTICS_ID": "xxxx", "ANALYTICS_SOSANALYTICS_REGION": "xxxx", "API_SOSWEBCONSOLEAPI_GRAPHQLAPIIDOUTPUT": "xxxx", "API_SOSWEBCONSOLEAPI_GRAPHQLAPIENDPOINTOUTPUT": "https://xxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql", "API_KEY": "", "HTTP_PROXY": "http://localhost:3128", "HTTPS_PROXY": "http://localhost:3128", "http_proxy": "http://localhost:3128", "https_proxy": "http://localhost:3128" }, "console": "integratedTerminal" } ] } More info: amplify: 3.10.0 node: v10.16.0 npm: 6.9.0

So, I tryied a lot of things but I dont know why it does work, Please help I getting crazy!

mooser commented 5 years ago

I bet you the reason it doesn't work is because the actual graphql endpoint you have configured by the time it runs data = await client.query({ query }); is not a valid endpoint.

Due to lambda pulling in the graphql endpoint from environment variables, and automatically generating the names of the variables, it is quite easy to get the endpoint wrong because it changed the name of an environment variable.

Cheers.

diego-palmeira commented 5 years ago

mm I dont understand your point. The error say 'request to https://xxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 52.85.219.177:443' that means the URL showed is the correct one I think, and if I try in a curl in the same VSC debug console, I get...

curl https://xxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql { "errors" : [ { "errorType" : "UnauthorizedException", "message" : "Valid authorization header not provided." } ] } It means there is connectivity from console to this endpoint, but not from amplify invoke command which hangs some minutes and then raise a TIMEOUT error... Have you some trick to set the proxy settings on amplify?

mooser commented 5 years ago

True, well you'll probably need to make a public repo to demonstrate the issue and send me link, because I can attest it does work if configured properly.

diego-palmeira commented 5 years ago

More info about my investigation:

1) After review the stacktrace it seems that my function calls --> aws-appsync --> apollo-client --> isomorphic-fetch and on this point it fails. It seems like proxy is not taken in consideration.

2) I review similar problems and may be it requires to use ProxyAgent similar to: https://github.com/apollographql/apollo-tooling/issues/523

3) I found a similar problem with amplify-cli at https://github.com/aws-amplify/amplify-cli/issues/459

4) Full stacktrace:

/home/user/.nvm/versions/node/v10.16.0/bin/node --inspect-brk=44994 ../../../../../../../usr/local/bin/amplify function invoke sosApiFunction [lrcc@lrcc-VirtualBox]:~/projects/aws/sos-project/src/sos (develop) $ /home/user/.nvm/versions/node/v10.16.0/bin/node --inspect-brk=44994 ../../../../../../../usr/local/bin/amplify function invoke sosApiFunction Debugger listening on ws://127.0.0.1:44994/3f9f8fdf-2d88-4d61-9918-31ea83ac0f51 For help, see: https://nodejs.org/en/docs/inspector Debugger attached. Using service: Lambda, provided by: awscloudformation ? Provide the name of the script file that contains your handler function: index.js ? Provide the name of the handler function to invoke: handler ? Provide the relative path to the event: event.json Testing function locally EVENT: {"httpMethod":"POST","body":"{\"name\": \"Amplify\"}","path":"/securityStatus","resource":"/{proxy+}","queryStringParameters":{}} App started

Success! Message:

{"statusCode":200,"body":"{\"success\":\"post securitystatus call succeed!\",\"url\":\"/securityStatus\",\"status\":[{\"statusCode\":\"OK\"}]}","headers":{"x-powered-by":"Express","access-control-allow-origin":"*","access-control-allow-headers":"Origin, X-Requested-With, Content-Type, Accept","content-type":"application/json; charset=utf-8","content-length":"102","etag":"W/\"66-i/N5I15byOX+ljlIp+yWRnOERS4\"","date":"Sat, 28 Sep 2019 18:35:51 GMT","connection":"close"},"isBase64Encoded":false} Missing x-apigateway-event or x-apigateway-context header(s) Promise { } { Error: Network error: request to https://xxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 143.204.229.90:443 at new ApolloError (/home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:85:32) at /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:1039:45 at /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:1411:21 at Array.forEach () at /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:1410:22 at Map.forEach () at QueryManager.broadcastQueries (/home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:1405:26) at /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/aws-appsync/node_modules/apollo-client/bundle.umd.js:988:35 at process._tickCallback (internal/process/next_tick.js:68:7) graphQLErrors: [], networkError: { FetchError: request to https://xxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 143.204.229.90:443 at ClientRequest. (/home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/isomorphic-fetch/node_modules/node-fetch/index.js:133:11) at ClientRequest.emit (events.js:198:13) at ClientRequest.EventEmitter.emit (domain.js:448:20) at TLSSocket.socketErrorListener (_http_client.js:392:9) at TLSSocket.emit (events.js:198:13) at TLSSocket.EventEmitter.emit (domain.js:448:20) at emitErrorNT (internal/streams/destroy.js:91:8) at emitErrorAndCloseNT (internal/streams/destroy.js:59:3) at process._tickCallback (internal/process/next_tick.js:63:19) name: 'FetchError', message: 'request to https://xxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 143.204.229.90:443', type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }, message: 'Network error: request to https://xxxx.appsync-api.eu-central-1.amazonaws.com/graphql failed, reason: connect ETIMEDOUT 143.204.229.90:443',

5) Is it possible aws-appsync needs to include the proxy-agent in some way?

Thank you for your help!, I need to unblock this problem, any help or idea is welcome!

diego-palmeira commented 5 years ago

Hi again, After modified /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/isomorphic-fetch/node_modules/node-fetch/index.js file to include:

At the begining of the file // Add proxy support import proxyAgent from 'http-proxy-agent';****

and harcoding proxy for test (near line 120)... ... // http.request only support string as host header, this hack make custom host header possible if (options.headers.host) { options.headers.host = options.headers.host[0]; }

// Add proxy support options.agent = new proxyAgent('http://localhost:3128');

// send request var req = send(options); var reqTimeout; ... then go to the isomorphic-fetch module main folder (in my case: /home/user/projects/aws/sos-project/src/sos/amplify/backend/function/sosApiFunction/src/node_modules/isomorphic-fetch) and run to install the dependency. sudo npm i http-proxy-agent

After that,

It seems the calls go out and reached the endpoint !, it does not hangs (with timeout) on "promise call" anymore, it is called immediately through my local cntlm proxy. Unfortunately, I'm facing other error reponsed by the cloudfront endpoint 400 bad request, maybe caused by permissions or something else that I have to review now. But the important point here is that isomorphic-fetch apparently does not support run behind a local proxy !

Any idea about how to implement it in a more elegant way?

thank you,

diego-palmeira commented 5 years ago

Finally, I use the agent in my lambda to add the proxy programatically using the library const https = require('https'); and forgetting the example of apollo.

Then in the lambda code: ..... const req = new AWS.HttpRequest(appsyncUrl, region); req.method = "POST"; req.headers.host = endpoint; req.headers["Content-Type"] = "application/json"; req.body = JSON.stringify({ query: graphqlQuery, operationName: "[your grapthql operation e.g. ListTodos]" }); // Add proxy support const localProxy = process.env.LOCAL_PROXY; if(localProxy) { var proxyAgent = require('proxy-agent'); var proxy = new proxyAgent(localProxy); AWS.config.update({ httpOptions: { agent: proxy } }); req.agent = proxy; } ....

That's all. BTW: I've followed the full example at https://aws-amplify.github.io/docs/cli-toolchain/quickstart#graphql-from-lambda, but adding the proxy as above to run it in my local, nad it works now.

We can close this ticket I think. thank you

github-actions[bot] commented 3 years ago

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels for those types of questions.