tricorder-observability / Starship

Starship: next-generation Observability platform built with eBPF+WASM
GNU Affero General Public License v3.0
163 stars 25 forks source link

API Server has lots of slow query into SQLite #28

Closed nascentcore-eng closed 1 year ago

nascentcore-eng commented 1 year ago

image

Just deploy Starship and check API server's log

nascentcore-eng commented 1 year ago

DB file is stored pv, persistent volume, which is backed by remote storage.

Need to reproduce on a kubernetes cluster. Just running API server is not going to reproduce this.

zh9310 commented 1 year ago

csi driver has limit iops, maybe this slow query .

jmzwcn commented 1 year ago

SQLite don't support list/watch as etcd, the following code will loop too quickly:

https://github.com/tricorder-observability/starship/blob/main/src/api-server/grpc/deployer.go#L52

So I think we could slower it manually.

tricorder-eng commented 1 year ago

We have chatted yesterday, and decided that we'd want to create a channel between API server's http component and gRPC component.