neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.1k stars 439 forks source link

Avoid extra memcopy per block when building SLRU segment #6434

Open VladLazar opened 9 months ago

VladLazar commented 9 months ago

Seems to me like instead of extend_from_slice (= effectively memcopying the returned page images into self.buf), we could make self.buf store the Bytes instead, and consume them in self.flush().

That optimization saves 1 memcopy.

_Originally posted by @problame in https://github.com/neondatabase/neon/pull/6372#discussion_r1461882184_

VladLazar commented 9 months ago

We can skip the intermediate buffer by getting the underlying writer from the archive builder (get_mut) and writing straight to it. We'd have to be careful to apply the same padding logic as tokio-tar.

VladLazar commented 9 months ago

Could do the much cleaner solution below instead:

turn the Vec into a stream with futures::stream::from_iter, then use tokio_util::io::StreamReader to expose that as tokio::io::Read to Archive::append.

https://github.com/neondatabase/neon/pull/6372#discussion_r1462275014