search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.11k stars 439 forks source link

noisy `Oversized vectored read (131100 > 131072) for keys` in prod logs #8967

Closed problame closed 1 month ago

problame commented 2 months ago

Sample

synthetic_size_worker:calculate_synthetic_size{tenant_id=$TENANT shard_id=0000}:gather_size_inputs{tenant_id=$TENANTshard_id=0000}:find_gc_cutoffs{timeline_id=$TENANT}: Oversized vectored read (131100 > 131072) for keys 01000000000000000100000006000000001F@F/B10B6FB8
VladLazar commented 2 months ago

Don't think it pug per say, but will take a look.

problame commented 2 months ago

One more occurence with a different oversize

2024-09-09T10:38:00.505829Z  WARN initial_size_calculation{tenant_id=$TENANT_ID shard_id=0000 timeline_id=$TIMELINE_ID}:logical_size_calculation_task: Oversized vectored read (165376 > 131072) for keys 000000067F00004002000000000000000001@32/84139CE0
VladLazar commented 2 months ago

We print this when trying to read a key value that's larger than 128KiB. Wondering why those values are that big, but we can bump this soft limit up to say 256KiB if we are okay with that. Alternatively, we can downgrade the log line.

koivunej commented 1 month ago

000000067F00004002000000000000000001

$ cargo run -qp pagectl key 000000067F00004002000000000000000001
parsed from hex: 000000067F00004002000000000000000001:

Key { field1: 0, field2: 1663, field3: 16386, field4: 0, field5: 0, field6: 1 }
rel_block:         false
rel_vm_block:      false
rel_fsm_block:     false
slru_block:        false
inherited:         true
rel_size:          false
slru_segment_size: false
recognized kind:   Some(RelDir(1663/16386))

So, maybe there is a lot of relations or a large relation?

VladLazar commented 1 month ago

Only eu-central-1 is warning about this consistently.

There's three problematic tenant/timelines:

000000067F00006000000000000000000001 is a reldir and 01000000000000000100000006000000001F is an SLRU block. Pageserver doesn't have control for either of these.

VladLazar commented 1 month ago

Action items:

VladLazar commented 1 month ago

From chat with Heikki: it's okay for slru deltas to be slightly above 128KiB due to overhead in the storage format. Let's bump the threshold to 130KiB and allow list unbounded key types (reldir & dbdir).