CompressionType

[zengqingfu1442]

How add a new compression type such as gzip or openssl for rust-rocksdb?And then how to build it for the TiKV storage engine?Thanks.

[siddontang]

Hi @zengqingfu1442

The compression type must be already supported in RocksDB, so you should first support this in RocksDB.

[zengqingfu1442]

Yes, I know. It is supported in RocksDB now and installed it on my machine. But should I rewrite it with rust language?

[siddontang]

You just need to add them in https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/crocksdb/crocksdb/c.h#L949 and https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/src/lib.rs#L87 if RocksDB supports it.

[zengqingfu1442]

Thanks, but how can I build the TiKV with the new storage engine RocksDB supporting new compression type ?

[siddontang]

you can build rocksdb dynamically, use your own rust-rocksdb in https://github.com/pingcap/tikv/blob/master/Cargo.toml#L78 then run make release.

[zengqingfu1442]

Thanks, I see. Should I also put the source code of rocksdb which I have modified in my own rust-rocksdb repository? And put where?

[siddontang]

TiKV only needs linking your RocksDB lib dynamically, it doesn't care the source.

BTW, if you want to link your own RocksDB statically, you can update https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/build.sh#L127

[zengqingfu1442]

Thanks, I see. Now, i have used tidb-ansible to deploy a TiDB cluster(including TiDB server, PD server and TiKV server), should I install my own RocksDB on each node?

[siddontang]

1. If you let TiKV link RocksDB dynamically, you should install on each node.
2. If you build RocksDB statically in TiKV, you don't need to do it.

[zengqingfu1442]

Thanks, but why don't rocksdb for TiKV use JEMALLOC when compiling? https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/build.sh#L138

[siddontang]

/cc @BusyJay

[BusyJay]

Allocation will be taken over by rustc when building a binary, so link to jemalloc is unnecessary.

[zengqingfu1442]

Thanks, Jay Lee. Can use the librocksdb_sys/build.sh other than build.rs written in librocksdb_sys/Cargo.toml when compiling rust-rocksdb? @BusyJay

[BusyJay]

I don't get it. Do you mean you want to use librocksdb_sys/build.sh to build a native rocksdb library? I think it should work but it's recommended to follow the instruction in rocksdb repository instead.

[zengqingfu1442]

Thanks, I now have built the tikv link rocksdb dynamically supporting a new compression type. I used the rocksdb5.7.0 though the code in https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/crocksdb/c.cc#L40 only support 5.7.3+, but I changed the 3 to 0, and tikv built successfully. What problem will it happen or come across using rocksdb5.7.0?thanks

[siddontang]

I remember that 5.7.3+ fixes some bugs, but I think it doesn't matter in your test.

But If you want to use in production, I suggest using a newer one.

[zengqingfu1442]

Thanks, I have integrated a new compression which is supported by rocksdb5.7.0 into tikv-v1.0.1, but now I use tikv-v1.0.0, it come across this error when compilation, it's the file "tikv/src/config.rs"

Compiling tikv v1.0.0 (file:///home/dzeng/tidb/build-tidb/deps/tikv) error[E0061]: this function takes 4 parameters but 1 parameter was supplied --> src/config.rs:92:39 | 92 | block_base_opts.set_lru_cache($opt.block_cache_size.0 as usize); | _____^ 93 | | block_base_opts.set_cache_index_and_filter_blocks($opt.cache_index_and_filter_blocks); 94 | | block_base_opts.set_pin_l0_filter_and_index_blocks_in_cache( 95 | | $opt.pin_l0_filter_and_index_blocks); ... | 157 | | pub fn build_opt(&self) -> ColumnFamilyOptions { 158 | | let mut cf_opts = build_cfopt!(self); | | -----------------^- | ||____| | | expected 4 parameters | in this macro invocation ....

[siddontang]

@zengqingfu1442

TiKV doesn't use the latest rust RocksDB, we will update it later. Now you can complement the missing args by yourself.

[zengqingfu1442]

The default compression of tidb is snappy,right? So, if I wanna use another compression type, should I set the parameter related to compression in tidb-ansible/conf/tikv.yml before deploying tidb?

[siddontang]

No, the default is lz4 and zstd for different levels.

You must update with your own type before deploying.

[zengqingfu1442]

Thanks, it's in tidb-ansible/conf/tikv.yml to update?

[siddontang]

yes, you must also put your tikv-server in the resource/bin

[zengqingfu1442]

If I put the my built tikv-server in tidb-ansible/downloads, will it work? Must I put it under tidb-ansible/resource/bin? Thanks.

[siddontang]

tidb-ansible/resource/bin then run rolling update.

[zengqingfu1442]

I tried to put my built tikv-server,tikv-ctl,tikv-fail into tidb-v1.0.1.tar.gz which replaced the original tikv,and then deploy, at last, I can also get the same tikv- binary under the deploy directory (just need to copy the share_lib *.so to preper directory).

[zengqingfu1442]

I wiil also put the built tikv binaries to tidb-ansible/resources/bin to replace the original binary. Thanks.

[zengqingfu1442]

What's the meaning of the compression-per-level?How can I use the specific compression type in rocksdb defaultcf?

[siddontang]

you must first know rocksdb konws your compression, and then you can set the compression name for the level you want to compress.

[zengqingfu1442]

Yes, I have let rocksdb know it, and tikv-server, tikv-ctl have also linked to the dynamical linking library of the compression.

[zengqingfu1442]

Now I have let rocksdb know the new compression, but tikv doesn't know it. I have put the new tikv binary under resources/bin and then rolling_update it.When I use tidb-ansible to start the tidb cluster, tikv always can't start up. The tikv_stderr.log says: invalid configuration file "conf/tikv.toml": invalid value: string "csszlib", expected invalid compression type for key rocksdb.defaultcf.compression-per-level

The new tikv binary has been able to linked to the dynamical linking library of the new compression. [dzeng@dzeng bin]$ ldd tikv-server linux-vdso.so.1 => (0x00007ffc89bf0000) librocksdb.so.5.7 => /usr/local/lib/librocksdb.so.5.7 (0x00007f8960065000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f895fd5c000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f895fb58000) librt.so.1 => /usr/lib64/librt.so.1 (0x00007f895f950000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f895f733000) libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f895f51d000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007f895f15a000) /lib64/ld-linux-x86-64.so.2 (0x000055a81029f000) libm.so.6 => /usr/lib64/libm.so.6 (0x00007f895ee57000) libsnappy.so.1 => /usr/lib64/libsnappy.so.1 (0x00007f895ec51000) libgflags.so.2.1 => /usr/lib64/libgflags.so.2.1 (0x00007f895ea30000) libz.so.1 => /usr/lib64/libz.so.1 (0x00007f895e819000) libcssz.so => /usr/lib64/libcssz.so (0x00007f895e571000) libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007f895e361000) libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f895e154000) libtbb.so.2 => /usr/lib64/libtbb.so.2 (0x00007f895df1f000)

How can I let tikv know it when useing tidb-ansible to depoly the cluster? Should I also modify some code in tidb-ansible?Thanks.

[siddontang]

Do you add your compression type here https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/src/lib.rs#L88 and here https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/crocksdb/crocksdb/c.h#L956?

Does your RocksDB already know your compression type?

[zengqingfu1442]

Yes, I have added and use the modified rust-rocksdb to compile tikv successfully. The rocksdb have already know the compression.

[zengqingfu1442]

Now, I think that maybe I should change the rocksdb defaultcf compression type in tikv/src/config.rs from lz4 or zstd to the new compression before compiling tikv:

impl Default for DefaultCfConfig { fn default() -> DefaultCfConfig { DefaultCfConfig { block_size: ReadableSize::kb(64), block_cache_size: ReadableSize::mb(memory_mb_for_cf(false, CF_DEFAULT) as u64), cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: true, whole_key_filtering: true, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, read_amp_bytes_per_bit: 0, compression_per_level: [ DBCompressionType::No, DBCompressionType::No, DBCompressionType::Lz4, DBCompressionType::Lz4, DBCompressionType::Lz4, DBCompressionType::Zstd, DBCompressionType::Zstd, ],

[siddontang]

you should add your compression type here too https://github.com/pingcap/tikv/blob/master/src/util/config.rs#L52

[zengqingfu1442]

I have done it. Otherwise, the compilation should raise error.

[zengqingfu1442]

As well as here https://github.com/pingcap/tikv/blob/master/src/util/rocksdb/mod.rs#L40

[zengqingfu1442]

How can we check what compression type the tikv support? In rocksdb, after make release, we can use binary db_bench or db_sanity_test to run a sanity test checking what compression the rocksdb supports, ./db_sanity_test /mnt/sfx-card-root/rocksdb570/ create ./db_sanity_test /mnt/sfx-card-root/rocksdb570/ verify

[siddontang]

you can add your compression type to the fast compression directly, like

const COMPRESSION_PRIORITY: [DBCompressionType; 4] = [
    DBCompressionType::Lz4,
    DBCompressionType::Snappy,
    DBCompressionType::Zstd,  
    your
];

[zengqingfu1442]

Yes, I have done before.

[siddontang]

what is the compilation error now?

[zengqingfu1442]

Can I change the number of supported compression type from 7 to 10, which including 7 default compression as well as csszlib(my new compression type), ZstdNotFinal, Disable ? I think this should not raise error in tikv.log or tikv_stderr.log when I set the new compression type in tidb-ansible/conf/tikv.yml and then deploy and start.

[zengqingfu1442]

No compilation error now. I have built tikv with new compression successfully.

[zengqingfu1442]

But if I specify the compression type in tidb-ansible/conf/tikv.yml with the new compression type, rocksdb.defaultcf: compression-per-level: ["no", "no", "csszlib", "csszlib", "csszlib", "snappy", "snappy"] rocksdb.writecf: compression-per-level: ["no", "no", "csszlib", "csszlib", "csszlib", "snappy", "snappy"] raftdb.defaultcf: compression-per-level: ["no", "no", "csszlib", "csszlib", "csszlib", "snappy", "snappy"] Then ansible-playbook start.yml won't start tikv, the tikv_stderror.log will say "invalid string value csszlib".

But when I don't specify it in tidb-ansible/conf/tikv.yml and leave ti as default, tikv can be started up, and the tikv.log will show that the tikv recognized the new compression.

[zhangjinpeng87]

please use zlib not csszlib

[siddontang]

@zengqingfu1442

You said that you have already fixed the problem, why still meet "invalid string value" error? Do you build your TiKV and put to resource/bin again?

[zengqingfu1442]

Yes, I have fixed the problem. I didn't change the default number of compaction level which is 7 in the tikv/src/config.rs, so when I start the tikv using ansible-playbook start.yml, the binary still just recognize the default 7 compression(no, snappy, zlib, bzip2, lz4, lz4hc, zstd) though the default compression-per-level has changed to [no,no,csszlib,csszlib,csszlib,snappy,snappy].

Now, I have changed the number of compression level from 7 to 10, including no,snappy,zlib,bzip2, csszlib, lz4, lz4hc, zstd, zstdnotfinal, disable. After building, I put the new binary tikv-* under tisb-ansible/resources/bin, then I change(uncomment) the compression-per-level in tidb-ansible/conf/tikv.yml from 7 compression for 7 levels to 10 compression for 10 levels. If I don't uncomment the compression-per-level , the default compression for 10 levels is just decided by the code
in /tikv/src/config.rs which is changed by me to [no,no, csszlib,csszlib,csszlib,csszlib,csszlib,csszlib,snappy,snappy]. Thanks.

[zengqingfu1442]

Is there any way or tools to run sanity test checking what compression type the built tikv-server support? In rocksdb, we can use ./db_sanity_test to check.

[siddontang]

@zengqingfu1442

We have no way now, maybe you can help us write a rust tool.

[zengqingfu1442]

One of the three tikv instances is down, and the tikv.log show these messages:(there is no errors in tikv_stderr.log). Can you help me analysis the errors?Thanks.

2017/11/28 13:10:03.476 tikv-server.rs:122: [INFO] start prometheus client 2017/11/28 13:10:03.476 mod.rs:209: [INFO] starting working thread: split check worker 2017/11/28 13:10:03.476 tikv-server.rs:234: [INFO] start storage 2017/11/28 13:10:03.477 mod.rs:209: [INFO] starting working thread: snapshot worker 2017/11/28 13:10:03.478 mod.rs:209: [INFO] starting working thread: raft gc worker 2017/11/28 13:10:03.480 mod.rs:209: [INFO] starting working thread: compact worker 2017/11/28 13:10:03.485 future.rs:115: [INFO] starting working thread: pd worker 2017/11/28 13:10:03.485 region.rs:142: [INFO] [region 2] begin apply snap data 2017/11/28 13:10:03.486 mod.rs:209: [INFO] starting working thread: consistency check worker 2017/11/28 13:10:03.488 mod.rs:209: [INFO] starting working thread: apply worker 2017/11/28 13:10:03.497 region.rs:237: [ERROR] failed to apply snap: Other(StringError("Corruption: external file have corrupted keys"))!!! 2017/11/28 13:10:03.508 mod.rs:209: [INFO] starting working thread: end-point-worker 2017/11/28 13:10:03.508 mod.rs:209: [INFO] starting working thread: snap-handler 2017/11/28 13:10:03.513 server.rs:155: [INFO] TiKV is ready to serve 2017/11/28 13:10:04.678 panic_hook.rs:99: [ERROR] thread 'raftstore-4' panicked '[region 2] 6 applying snapshot failed' at "src/raftstore/store/peer_storage.rs:1009" stack backtrace: 0: 0x7f0c1f42495e - backtrace::backtrace::libunwind::trace at /home/dzeng/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.2.3/src/backtrace/libunwind.rs:54

• backtrace::backtrace::trace at /home/dzeng/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.2.3/src/backtrace/mod.rs:70 1: 0x7f0c1f4250c3 - backtrace::capture::{{impl}}::new at /home/dzeng/tidb/integrate-csszlib-into-tidb-1.0.1/deps/tikv/target/release/build/backtrace-f2668981c6548436/out/capture.rs:79 2: 0x7f0c1f2b8766 - tikv::util::panic_hook::set_exit_hook::{{closure}}

[siddontang]

does your disk have any error? Do you add a TiKV and then it panics?