Tracking: Monitoring, logging and tooling improvements

[hzxa21]

As we have more and more users, we should start taking monitoring, logging and tooling seriously to make our life easier during debugging, especially when debugging live production issues. We can use this issue to track ideas on how to improve our logging and tooling in kernel (i.e. risectl).

Monitoring:

• [x] #8019
• [ ] Monitor abormal kv entry size in storage
• [x] #8035
• [x] #9708
• [x] Monitor operation conflict / sanity check failure rates in stateful opeartor
• [x] #8794
• [ ] #8795
• [x] #8814
• [x] #9725

Logging:

• [ ] Log table id, sst id, block id, epoch on all read/write errors
• [ ] Encode state table type in table id
• [ ] Log all historical DDL statements

Tooling:

• [x] #8020
• [x] Read-only access and tooling to different object store backends via risectl obj subcommand (without relying on RisingWave cluster to be up)
• [x] Read-only access and tooling to meta store (without relying on RisingWave cluster to be up)

[jon-chuang]

Monitor operation conflict / sanity check failure rates in stateful opeartor

What is this? Is it write conflict?

[hzxa21]

Monitor operation conflict / sanity check failure rates in stateful opeartor

What is this? Is it write conflict?

I was mainly refering to the memtable operation conflicts: https://github.com/risingwavelabs/risingwave/blob/d8198fa138003e1f1431053f4f5f09e4a5fa8fd8/src/storage/src/mem_table.rs#L96.

Write conflict is also a good metric but not neccessary critical if it is caused by DML.