Kong occasionally throws '404 - no Route matched with those values' in dbless mode

DanNguyen8189 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version (`$ kong version`)

Kong 2.8 dbless mode (with KIC 2.3.0)

Current Behavior

Once in a while, Kong starts returning 404 - no Route matched with those values for a route path that exists and continues before a manual kong restart in our production environments, which clears up the issue. Because of this, we have a scheduled cronjob to restart Kong every 12 hours to prevent this issue from happening. However, we'd like to remove this.

There's no noticeable change in traffic volume to Kong or that specific route during the incident. No changes to ingress objects configurations or services that we know of. Our kong dashboard displays no correlation with latency, upstream time, bandwidth, etc during time of incident. Kong pod logs do not reveal anything related to 404s.

We've been unable to reproduce this scenario in staging environments. This is an ongoing issue with incidents in the past that also happened with an older kong version (2.3). The only way we've been able to get 404s to specific routes in staging was by reducing the mem_cache_size value very small to the point where the routing configuration doesn't fit in the allocated memory anymore. We think that at some point, a small part of the routing configuration is lost from the proxy resulting in one route returning 404s. Restart might be clearing the issue because the ingress-controller gets to write the whole routing configuration back in. Unsure of cache could also be culprit.

Number of replicas and memory size has been increased in the past and appears to reduce the frequency but does not solve it

Expected Behavior

No response

Steps To Reproduce

We haven't been able to reproduce this exact scenario in staging environments, but here are some of out settings at the times of last incident and we can elaborate more if necessary: Kong 2.8 KIC 2.3.0 (this has since been upgraded to 2.4.2) database: "off" mem_cache_size: "2048m" (this has since been changed to "6144m") nginx_worker_processes: "1" nginx_proxy_large_client_header_buffers: "4 32k" nginx_proxy_add_header: 'Strict-Transport-Security "max-age=31536000; includeSubDomains" always' nginx_http_log_format: (too big to fit here, not important) (other nginx settings are default/unchanged) minReplicas: 4 Kong pod memory sizing

  resources:
    limits:
      cpu: 4
      memory: 8Gi
    requests:
      cpu: 4
      memory: 8Gi

KIC memory sizing:

    resources:
      limits:
        cpu: 4
        memory: 8Gi
      requests:
        cpu: 4
        memory: 8Gi

Our kongs ran on AWS eks clusters version 1.20 (they have since been upgraded to 1.21) Total traffic goes up and down between 50 and 250 requests per second We use the prometheus plugin with default settings

Anything else?

No response

StarlightIbuki commented 1 year ago

Thanks for reporting this. I'm not sure if I understand the issue correctly. Is the 404 happening right after we configure a new route? Or did it just happen randomly? Ping @randmonkey to take a look.

cwoolsey-nice-cxone commented 1 year ago

Thanks for reporting this. I'm not sure if I understand the issue correctly. Is the 404 happening right after we configure a new route? Or did it just happen randomly? Ping @randmonkey to take a look.

We are seeing a very similar issue running Kong 2.8.1, but we're not running dbless mode. In our case, it's not related to configuring new routes, it will just start happening randomly. It's also very inconsistent, the same request might succeed 4 times, but fail 2 of the times. It almost seems like it's an issue at the worker process level, though we don't have a good way to confirm that.

StarlightIbuki commented 1 year ago

Thanks for reporting this. I'm not sure if I understand the issue correctly. Is the 404 happening right after we configure a new route? Or did it just happen randomly? Ping @randmonkey to take a look.

We are seeing a very similar issue running Kong 2.8.1, but we're not running dbless mode. In our case, it's not related to configuring new routes, it will just start happening randomly. It's also very inconsistent, the same request might succeed 4 times, but fail 2 of the times. It almost seems like it's an issue at the worker process level, though we don't have a good way to confirm that.

In this case, is Kong also running with KIC?

cwoolsey-nice-cxone commented 1 year ago

Thanks for reporting this. I'm not sure if I understand the issue correctly. Is the 404 happening right after we configure a new route? Or did it just happen randomly? Ping @randmonkey to take a look.

We are seeing a very similar issue running Kong 2.8.1, but we're not running dbless mode. In our case, it's not related to configuring new routes, it will just start happening randomly. It's also very inconsistent, the same request might succeed 4 times, but fail 2 of the times. It almost seems like it's an issue at the worker process level, though we don't have a good way to confirm that.

In this case, is Kong also running with KIC?

No. In our case we are only running Kong.

StarlightIbuki commented 1 year ago

Internal ticket created: KAG-2729

kikito commented 1 year ago

@DanNguyen8189 I am afraid we can't dedicate effort to investigate an issue like this in an older version of Kong (2.8, in this case).

We think that at some point, a small part of the routing configuration is lost from the proxy resulting in one route returning 404s.

Unfortunately that doesn't narrow the issue enough.

Here's some recommendations:

If you are using any custom plugins, please try deactivating them and seeing if the problem persists. Sometimes custom plugins can produce unexpected results like this.
The fact that this happens in production but not on staging is highly suspicious.
Instead of a full restart, perhaps a config refresh (sending the same configuration the that gateway already has) might be good enough.
Try to upgrade to 3.4 and see if the problem persists. If it does, we should be able to investigate it for 3.4. The 3.x series has many changes compared with 2.8, one of which being less reliance on shared dictionaries / caches in favor of LMDB.

github-actions[bot] commented 1 year ago

This issue is marked as stale because it has been open for 14 days with no activity.

VonsName commented 12 months ago

we met the issue yet

github-actions[bot] commented 11 months ago

This issue is marked as stale because it has been open for 14 days with no activity.

github-actions[bot] commented 11 months ago

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team

Kong / kong