pageserver: optimize `HashMap::insert()` in `SerializedValueBatch::extend()`

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

15.22k stars 444 forks source link

pageserver: optimize `HashMap::insert()` in `SerializedValueBatch::extend()` #9848

Closed erikgrinaker closed 7 hours ago

erikgrinaker commented 7 hours ago

During bulk ingestion in the WAL receiver, HashMap::insert shows up prominently in profiles under SerializedValueBatch::extend via validate_lsn_order. This likely contributed to pathologically slow bulk ingestion in this Slack thread.

profile.svg.gz

erikgrinaker commented 7 hours ago

Regression caused by https://github.com/neondatabase/neon/pull/9579. Profile on the parent doesn't show the hashmap inserts:

profile-before.svg.gz

hlinnaka commented 7 hours ago

Hmm, all the validate_lsn_order calls are guarded by if cfg!(any(debug_assertions, test)). Are you sure you used a release build?

I don't actually see validate_lsn_order in the flamegraphs you included though.

erikgrinaker commented 7 hours ago

Oof, you're right. Was just blindly following the profiles. This is a red herring then.

I don't actually see validate_lsn_order in the flamegraphs you included though.

Should be in the link on the main comment: https://github.com/user-attachments/files/17868666/profile.svg.gz