API Server has lots of slow query into SQLite

tricorder-observability / Starship

Starship: next-generation Observability platform built with eBPF+WASM

GNU Affero General Public License v3.0

163 stars 25 forks source link

Closed nascentcore-eng closed 1 year ago

nascentcore-eng commented 1 year ago

Just deploy Starship and check API server's log

nascentcore-eng commented 1 year ago

DB file is stored pv, persistent volume, which is backed by remote storage.

Need to reproduce on a kubernetes cluster. Just running API server is not going to reproduce this.

zh9310 commented 1 year ago

csi driver has limit iops, maybe this slow query .

jmzwcn commented 1 year ago

SQLite don't support list/watch as etcd, the following code will loop too quickly:

So I think we could slower it manually.

tricorder-eng commented 1 year ago

We have chatted yesterday, and decided that we'd want to create a channel between API server's http component and gRPC component.

When http component receives user requests of any kind
Send a message over the channel
The gRPC component wait on the channel, and perform a full query over the sqlite db about (to be deployed module, and to be underployed, and to be deleted etc)