feat(storage): do not block ingest_batch when buffer is full

[Little-Wallace]

Problem

To avoid compute-node crash down caused by the increasing memory of shared-buffer, we will flush shared-buffer when the memory of which reaches capacity. See more details in https://github.com/singularity-data/risingwave/pull/2036.

There is another problem is that this operation will block ingest_batch calling by executor a long time. Because there is only one upload thread, the throughput of our system is decided by the throughput of upload thread.

Solution

1. increase capacity of shared buffer.
2. start ingesting new batch immediately after moving buffer from non_upload_batches to uploading_batches, do not wait upload task finished.
3. uploading files generated by shared buffer in parallel.

[Little-Wallace]

cc @skyzh This problem may cause your load task fail

[Little-Wallace]

If let ingest_batch blocking when buffer reaches capacity is a feature by designed, we shall increase the capacity to at least three times of sstable_size , which sets 256 MB by default. So that we can upload data to s3 in parallel.

[wenym1]

I have a rough plan for this.

We can take a high and low threshold. When the memory usage reach the low threshold, we launch an upload task to uploader but do not block the ingest_batch call. Only when the memory usage reaches the high threshold we block the ingest_batch call.

Another feasible improvement will be spawning more uploader threads.

[Little-Wallace]

@wenym1 Agree! Would you mind refactoring it?

[wenym1]

Would you mind refactoring it?

Sure. I am currently working on it.