Express Gateway fails when invalid Service Endpoint is configured

[surykv]

Hi,

We were trying to configure API end points, Service end points and pipelines in Express Gateway.

By mistake, if we configure an invalid Service Endpoint, the whole express gateway just hangs with exception trace below - (node:1) UnhandledPromiseRejectionWarning: Error: service endpoint http://api.loan-test.com/v1/products (referenced in proxy policy configuration) does not exist at module.exports (/usr/src/app/lib/policies/proxy/proxy.js:24:11) at policyOptions.policy.args (/usr/src/app/lib/policies/index.js:15:14) at policyStep (/usr/src/app/lib/gateway/pipelines.js:161:32) at arrayMap (/usr/src/app/node_modules/lodash.flatmap/index.js:140:21) at map (/usr/src/app/node_modules/lodash.flatmap/index.js:1928:10) at flatMap (/usr/src/app/node_modules/lodash.flatmap/index.js:1881:22) at pipelinePoliciesConfig.forEach.policyConfig (/usr/src/app/lib/gateway/pipelines.js:146:25) at Array.forEach () at configurePipeline (/usr/src/app/lib/gateway/pipelines.js:130:26) at Object.module.exports.bootstrap (/usr/src/app/lib/gateway/pipelines.js:23:32) (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)

Is there a way of gracefully handling this? There might be instances where by mistake a typo or something is present in the Service Endpoint URL.

We are in the mid of a production release. But this error is scary.

[XVincentX]

This should not scary you at all; Express Gateway validates anything before starting the software and will refuse invalid configuration, but won't stop running. It will simply refuse the new file while retaining the old one in memory and use it.

[surykv]

@XVincentX Thanks for your reply. I'll explain a little bit about our use case. We have an app that creates these configurations in Express Gateway through Admin APIs (which is again a pipeline in EG). Let's say the app creates the service endpoint without knowing that its invalid. Now express gateway doesn't throw any error during the actual creation but later when the pipeline is created, this exception is thrown in background. How can the app know that the express gateway is invalid?. Or should the app do service endpoint validations before creating them in express gateway? Or should express gateway throw invalid config errors synchronously when pipeline is being created?

Also, we have observed because of this exception, the redis is being affected and Express Gateway is throwing 502 errors. Also, its not possible to deactivate such end points through Admin APIs since that also goes via EG and its not reachable at that time. We fixed it by entering the container and removing entry from gateway.config.yml.

[XVincentX]

@surykv This is kind of weird, this should not be the behaviour and if it is, I'll fix it.

We have an app that creates these configurations in Express Gateway through Admin APIs (which is again a pipeline in EG).

So basically I'm assuming you're proxying requests using Express Gateway itself into the Admin API, right? That's ok.

How can the app know that the express gateway is invalid?

When you're doing the API call, in case the new pipeline fails to get inserted in Express Gateway you should receive an error status code, and not a 200; did you verify that?

the redis is being affected and Express Gateway is throwing 502 errors

This is interesting. Is there any way to reproduce this error?

[surykv]

@XVincentX

When you're doing the API call, in case the new pipeline fails to get inserted in Express Gateway you should receive an error status code, and not a 200; did you verify that?

No, it did not fail. It gave a 200 and it inserted the pipeline into the gateway.config.yml. The service endpoint when I created it gave a 200, I used the service endpoint name and created the pipeline it gave a 200, but the exception was being thrown and we could see in the logs and the Express Gateway stopped accepting requests. To add to this, we are running Express Gateway in a docker container, the EG_CONFIG_DIR is /var/lib/eg which is pointing to a docker volume in the docker stack file.

[XVincentX]

It would be great to create a minimal repo with an example; this should definitely not be happening.

[surykv]

@XVincentX Let me know if you need any help from me in doing this.

[XVincentX]

I think if you could write me all the steps to reproduce this error I could look into and see what’s going on.

On 30 Sep 2019, at 11:32, surykv notifications@github.com wrote:

@XVincentX Let me know if you need any help from me in doing this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[surykv]

1) Deployed the express Gateway as a docker container 2) Mapped /var/lib/eg folder to a volume (so that the gateway config yml is persistent across container restarts) 3) Attached the gateway.config.yml file we use (where admin API is also a pipeline) gateway.config.txt

4) Created an API Endpoint using below request: {{eg_url}}/api-endpoints/testing-endpoint Body: { "host": "", "paths":["/api/v1/"], "scopes": ["test"] } Response 201

5) Created the service endpoint as below {{eg_url}}/service-endpoints/testing-service-endpoint Body: { "url":"http://api.loan-test.com/v1/products" } Response 201

6) Created the pipeline as below: {{eg_url}}/pipelines/testing-pipeline Body: { "apiEndpoints": [ "testing-endpoint" ], "policies": [ { "key-auth": [ { "action": { "apiKeyHeaderScheme": "Bearer" } } ] }, { "proxy": [ { "action": { "changeOrigin": true, "serviceEndpoint": "testing-service-endpoint" } } ] } ] } Response: 201

All these values got updated in gateway.config.yml. After that, the exception started getting thrown, we could see it in log files and Express Gateway stopped accepting requests. We couldn't disable the invalid config also because EG was not accepting any API request

@XVincentX

[surykv]

@XVincentX Any update on this issue?

[XVincentX]

I haven't been able to check this out yet.

Can you properly format the code though? Otherwise it's a pain to copy paste and reproduce.