tokio-epoll-uring: panic in prod due to tokio-epoll-uring creation failure

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.78k stars 429 forks source link

tokio-epoll-uring: panic in prod due to tokio-epoll-uring creation failure #7136

Closed problame closed 7 months ago

problame commented 7 months ago

https://neondb.slack.com/archives/C033RQ5SPDH/p1710490501672359

### Tasks
- [x] ~~investigate why no sentry message to slack channel~~ decided not to pursue this
- [ ] https://github.com/neondatabase/neon/pull/7141
- [ ] https://github.com/neondatabase/neon/pull/7160
- [ ] https://github.com/neondatabase/neon/pull/7164
- [x] alert on launch failures

jcsp commented 7 months ago

It was an old kernel
We eliminated the code paths that use spawn_blocking + handle block_on
CHange last week: retry on out of locked memory instead of panicking.

Todo this week: create alert that uses new metrics