[RFC] Dirty Memory: CF memory limit vs WBM limit

[Yuval-Ariel]

A delay request that stems from memory usage can be sent to the WriteController from two sources - CF and WBM (when allow_stall and dynamic_delay are turned on). The CF memory limit is the max number of memtables and their sizes (max_write_buffer_number and write_buffer_size). Writes are delayed when writing to the last memtable (when max_write_buffer_number > 3) and completely stopped when reaching max_write_buffer_number. The WBM memory limit is its capacity and writes are delayed when memory_usage() exceeds start_delay_percent * capacity.

These two sources of delay writes might be problematic.

[Yuval-Ariel]

@hilikspdb @udi-speedb @Guyme plz discuss

[hilikspdb]

With enable speedb features we set the write buffer size and the number of write buffers to be high enough so the delay requests will only come from the WBM. This allow for a much better memory utilisation and much more gradual delay (many more steps). If the user wants to change staff she is on her own

[Yuval-Ariel]

why shouldnt we just ignore the delay requirements from the CF when theres also a DDM? increasing memtable size and number may have detrimental effects such as uneven L0 file size (W-amp increase) and big flush steps..

[hilikspdb]

if the user set limits she expect us to respect it. Unit tests may also fail etc. I am busy enough explaining behaviours