SST stats missing of compaction result of set global rocksdb_compact_cf="default"

[rockeet]

set global rocksdb_compact_cf = "default";

The compacted SST missing stats info distincts per prefix:

The correct SST stat info should like this:

We traced to Rdb_ddl_manager::safe_find returned nullptr to Rdb_tbl_prop_coll, which is caused by in Rdb_ddl_manager::init, Rdb_key_def is not fully initialized(Rdb_key_def::setup was not called). From this point, I have no idea about how to go forward and resolve this issue(Rdb_key_def::setup needs TABLE object which can not be obtained).

[Pushapgl]

Can you tell me the what is the value for rocksdb_table_stats_sampling_pct for your workload? There is sampling around the "distincts per prefix" which is added as part of https://github.com/facebook/mysql-5.6/commit/c3d6ffc2f96324cb0c436f1a6ed77441c8b3c16d.

Can you try to repro this after setting rocksdb_table_stats_sampling_pct = 0?

[rockeet]

mysql> show variables like 'rocksdb_table_stats_%';
+--------------------------------------------------+-------+
| Variable_name                                    | Value |
+--------------------------------------------------+-------+
| rocksdb_table_stats_background_thread_nice_value | 19    |
| rocksdb_table_stats_max_num_rows_scanned         | 0     |
| rocksdb_table_stats_recalc_threshold_count       | 100   |
| rocksdb_table_stats_recalc_threshold_pct         | 10    |
| rocksdb_table_stats_sampling_pct                 | 10    |
| rocksdb_table_stats_use_table_scan               | OFF   |
+--------------------------------------------------+-------+
6 rows in set (0.00 sec)

This issue just happens for non opened table, if a table is opened, stat of SSTs for that table are correct.

[Pushapgl]

Can you set rocksdb_table_stats_sampling_pct to 0, then see if the issue still exists?

[rockeet]

set rocksdb_table_stats_sampling_pct to 0 didn't resolve this issue.

I have read the code, this issue is not related to rocksdb_table_stats_sampling_pct.

[Pushapgl]

can you provide me a repro for the issue?

[rockeet]

Step 1

start a clean myrocks server

Step 2

run sysbench on a client

sysbench oltp_write_only prepare --threads=100 --tables=100 --table_size=100000 \
    --db-driver=mysql --mysql-host=<myrocks-server> --mysql-port=3306 \
    --mysql-user=test  --mysql-password=<passwd> --mysql-db=test \
    --mysql_storage_engine='rocksdb default charset latin1' 

Step 3

stop and restart myrocks server

Step 4

run mysql command on a client, NOTE: don't run any mysql command before this command.

set global rocksdb_compact_cf = "default";

Step 5

After compact completed, SST indexstat is wrong.

In our in house myrocks branch, we have a webview to see per-SST info thus we discovered this issue.

In myrocks world, I think this info should be added into information_schema.ROCKSDB_SST_PROPS as a json column.

[lth]

I've previously documented when we might get empty sst stats here:

https://github.com/facebook/mysql-5.6/issues/807

I think you are describing case #2

[rockeet]

@lth Yes, this is the issue you had documented. This issue is serious as it would yield wrong query plans, could you provide a fix?

[lth]

I don't have a good fix for you, but rocksdb_table_stats_use_table_scan was meant to address these problem, but we don't use it in production, so it is very possible that it has problems.