gp top breaks sometimes with too many requests to workspace info and never recover

gitpod-io / gitpod

The developer platform for on-demand cloud development environments to create software faster and more securely.

https://www.gitpod.io

GNU Affero General Public License v3.0

13.01k stars 1.24k forks source link

gp top breaks sometimes with too many requests to workspace info and never recover #11139

Closed akosyakov closed 2 years ago

akosyakov commented 2 years ago

Bug description

watch gp top fails at some point and does not rework in the workspace under load. I also had https://github.com/akosyakov/gitpod-monitor installed at the same time which make use of workspace info endpoint as well.

See internal thread.

gitpod /workspace/factorial-gitpod (master) $ gp top
2022/07/05 08:47:42 cannot get workspace resources: rpc error: code = Unknown desc = could not retrieve workspace info: rpc error: code = Internal desc = grpc: error while marshaling: proto: Marshal called with nil

Expectation

Running something like watch gp top should not fail even under load, since it is an only a way for a user to see perf problems. cc @Furisto

Steps to reproduce

See above.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

akosyakov commented 2 years ago

I think it happened because I also have https://github.com/akosyakov/gitpod-monitor which does requests each 2 second from each window. I think we need then caching and debouncing in supervisor, that a result is served from a cache within 1sec for instance.

kylos101 commented 2 years ago

Scheduled. @Furisto is on-call this week, so the earliest he'd get to peek is next week. @utam0k if you get spare bandwidth this week, perhaps you can look? Probably easiest to recreate if you run watch -n 1 gp top.

sagor999 commented 2 years ago

This is not an issue of load. It was simply rate limited as per design (as can be seen in screenshot: resource exhausted error). Reference: https://github.com/gitpod-io/gitpod/blob/ac58e182c2e7a985e1dee78deba03e07c405ecaa/components/workspacekit/cmd/rings.go#L1026-L1032 https://github.com/gitpod-io/gitpod/blob/ac58e182c2e7a985e1dee78deba03e07c405ecaa/components/ws-daemon/pkg/iws/iws.go#L187-L189

Current rate limit is 4 requests per 1.5s. Which I think is plenty?

How often does gp top and gitpod-monitor make requests? @akosyakov

utam0k commented 2 years ago

This problem could not be reproduced with watch -n 1 gp top . Is it still occurring in production?

sagor999 commented 2 years ago

You can easily reproduce this by opening 3 terminal windows and in each running watch -n 1 gp top. You will start seeing same error due to rate limit.

akosyakov commented 2 years ago

closing as duplicate of https://github.com/gitpod-io/gitpod/issues/12083