[INLONG-110] Optimize Broker storage to increase throughput

[dockerzhang]

I think the current Broker's read and write performance still has a relatively large room for improvement. We need to continue to iterate to improve the storage performance of the system. I have listed some considerations and hope to get some better suggestions:

1. Data read and write operations should consider the characteristics of the disk, for example, the disk is based on 512-byte sectors as its storage unit, and read data in batches of 64k; the file system will eliminate the cache according to certain rules Pages in memory etc. If the read and write operations take these contents into account, I believe that the current TPS can be higher;

2. Storage should consider the problem of fragmentation of disk space, such as pre-allocation of fixed-length files and reuse of aging files to enable continuous reading of disk files and improve data read and write speed;

3. The number of memory cache blocks should be configurable: the current memory cache is managed according to the fixed configuration of 2 memory blocks per topic. We should allow the business to build more memory cache space based on actual resource conditions;

4. More effective memory-to-disk operation: At present, the flashing operation is to flash messages from the memory to the disk one by one for storage. This block can be adjusted to write to the disk in batches according to the memory block, thereby improving storage efficiency;

5. Remove the SSD auxiliary consumption function: Because the SSD disk capacity is too small, the SSD storage consumption is not suitable for practical applications, so it should be removed to avoid user confusion, and related configurations and settings need to be cleaned up;

6. The stored file should increase the content of the file header, including the data version information, in order to facilitate the subsequent storage scheme is still seamlessly compatible with the data format of the old version;

7. Add CheckPoint check mechanism: the current system will only check the validity of the last file when it is restarted. In fact, when the system is shut down, there may be multiple consecutive files still in memory, the practice that only the last file is checked currently is easy to cause abnormal mixing into the data stream, we should add CheckPoint mechanism to improve this abnormal situation.

JIRA link - [INLONG-110] created by gosonzhang

[gosonzhang]

Closed due to long periods of inactivity

Please reopen it if needed