ydb-platform / nbs

Network Block Store
Apache License 2.0
52 stars 21 forks source link

[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

Closed qkrorlqr closed 5 months ago

qkrorlqr commented 7 months ago

We want a single FS to be able to provide more read throughput when more clients (VMs) connect to the FS. The idea is simple: right now the bottleneck for large reads is the IndexTablet, because all data is proxied via it. But we don't really need to transfer all data via the tablet. The tablet can return <BSGroupId, BlobId, ByteRange> tupes to the client (filestore-vhost) instead of returning the requested data. The client can read the data from the specified BSGroups by itself. A fallback should be implemented for the case when the client cannot read the data by itself - e.g. if the specified BlobId has already been deleted or if the client has no direct access to the storage nodes.

qkrorlqr commented 7 months ago

Включать фичу per fs будем как-то так: filestore-client executeaction --action changestorageconfig --input-json '{"FileSystemId": "cahh6fjsng36rp186gjq", "StorageConfig": {"TwoStageReadEnabled": true}}'

Проверять так: filestore-client executeaction --action getstorageconfigfields --input-json '{FileSystemId: "cahh6fjsng36rp186gjq", StorageConfigFields: ["TwoStageReadEnabled"]}'

qkrorlqr commented 7 months ago

Screenshot from 2024-02-19 22-02-36 managed to scale read throughput to 10 GB/s total for 10 VMs

debnatkh commented 7 months ago

After optimizations in IndexTablet managed to get 13-15 GB/s for 10 vms.

image

qkrorlqr commented 5 months ago

Implemented, deployed, works well, closing the issue

debnatkh commented 5 months ago

20 clients, fio, num_jobs = 32, bs = 1M

~25-30 GB/s

image