Move cleaned pages into read cache instead of discarding them.

[adamreichold]

Just after dirty pages have been written out to disk, they are currently discard and need to be read in again when these pages are accessed in the future.

This changes moves them into the read cache instead thereby keeping the clean data available and cache hot in memory.

It turned out that this was also what was making direct I/O slower in #808, i.e. this was not that noticeable when buffered I/O is use because the pages were read back in from the kernel page cache most of the time but it should be a win in both scenarios.

[adamreichold]

The intermediate fuzzer failure was me trying to be clever and avoid reference count bump by Option::takeing the buffers out of the write cache but that was not so clever if one of the intermediate writes would fail, I would never call clear and leave the empty buffers in place for the next flush or another call to write to assert on them. Should be fixed now.

[adamreichold]

The intermediate fuzzer failure was me trying to be clever and avoid reference count bump by Option::takeing the buffers out of the write cache but that was not so clever if one of the intermediate writes would fail, I would never call clear and leave the empty buffers in place for the next flush or another call to write to assert on them. Should be fixed now.

Turns out I can avoid that atomic reference count bump, but I think it is safer to just do this in two passes in any case.

[cberner]

Merged. Thanks!