Add docs for LSM

[purplefox]

Hi @jlerche thanks for this. I'm going to put this one on hold for now, as due the recent changes much of this is now out of date. Sorry for that, inevitable consequences of refactoring :(

I can see here you have documented a lot of stuff that comes before the LSM manager (which used to be called level manager and now lives in the lsm package). I wouldn't really consider that stuff to be part of the LSM.

In future docs for LSM it would be cool to the see the documentation concentrating on how the LSM manager works, including.

• Overview of how it works - key/value, most recent key retained
• Tombstones
• Versions
• What are levels, and why do we have them?
• What is special about L0?
• Maximum sizes of levels
• How do we choose a level to compact?
• How do we choose tables in a level to compact?
• How do we use tombstones to compact data?
• Prefix deletions
• Dead version ranges
• How do we scale compaction?
• How do we scale writes to L0?

[purplefox]

I'm going to close this for now.