apache / kvrocks

Apache Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol.
https://kvrocks.apache.org/
Apache License 2.0
3.53k stars 464 forks source link

Add a slow flag to commands with heavy disk IO #2474

Closed PragmaTwice closed 1 month ago

PragmaTwice commented 2 months ago

Search before asking

Motivation

We can add a new command flag e.g. slow and apply it to all commands that can lead to heavy disk IO (rocksdb read/write).

Solution

No response

Are you willing to submit a PR?

PokIsemaine commented 2 months ago

Is there any mechanism that will accompany this flag after it is added? And what is the metric for heavy disk IO?

mapleFU commented 2 months ago

Is there any mechanism that will accompany this flag after it is added?

Generally there're easier way and heavier way. Tagging can allow better techniques for scheduling, like independent io-task-pool or throttle for these tasks ( These techniques doesn't be implemented now, lol). Generally for user task "heavy request" might have a lower priority and avoid affect the light operations

And what is the metric for heavy disk IO?

No, lol, I guess this should be guessed by user and a bit hard to analysis, especially on zset or other heavy structure

PragmaTwice commented 2 months ago

And what is the metric for heavy disk IO?

A simple method is to check that if a command only involves a constant number of rocksdb read/writes, e.g. LPUSH, SET, HSET.

Otherwise it can be a time consuming operation, e.g. HGETALL, LINSERT... (the number of rocksdb read/writes is not constant, e.g. O(n), where n is a runtime variable like size of the data structure.)

PragmaTwice commented 2 months ago

Is there any mechanism that will accompany this flag after it is added?

Like mentioned by @mapleFU , it has potential benefits for optimization and scheduling.

Also, we can generate some doc from it to let users know which commands are time-consuming.

mapleFU commented 2 months ago

Besides, the heavy disk "write" IO api might have higher risk on new Context api: https://github.com/apache/kvrocks/pull/2332

jonathanc-n commented 2 months ago

Can I be assigned this issue?

git-hulk commented 2 months ago

@jonathanc-n Assigned, thank you!