[Feature]: PebbleMQ

[xiaofan-luan]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

Pebble is a KV storage written in go and aims to replace rocksdb in cockroach.

https://github.com/cockroachdb/pebble

We want to investigate the possibility of replace pebble with rocksdb so milvus can be more go native.

Some other benefits:

1. more easier cross platform support
2. maybe faster speed
3. better code readibility
4. better memory control

Describe the solution you'd like.

1. check if all features used in rmq is also supported with pebble
2. replace rmq with pebble
3. test the performance between rmq and pebble
4. make sure all files can be upgraded

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

keep it open

[vishjain]

/assign

[caesarjuly]

/assign

[xiaofan-luan]

Anyone interested please take it~

[caesarjuly]

Hi @xiaofan-luan , I'm looking into the previous PR. It's quite a huge PR. Since I'm still getting familiar with Milvus code, I'd like to know if it's better for me to replace the modules one by one - splitting the huge PR into several small PRs. As @jiaoew1991 shared before, I will keep the RocksDB code and add another implementation based on PebbleDB.

1. Add PebbleDB to the kv module and write basic tests
2. Add PebbleDB to mq module with the required tests
3. Add PebbleDB to configs and other modules
4. Do an integration test and verify the performance
5. Remove RocksDB from the dependency

Any suggestion is welcome :).

[xiaofan-luan]

• Add PebbleDB to the kv module and write basic tests
• Add PebbleDB to mq module with the required tests
• Add PebbleDB to configs and other modules
• Do an integration test and verify the performance
• Remove RocksDB from the dependenc

One thing I want to understand:

1. is pebble fully compatible with rocksdb? which is saying we can upgrade without change any data format?
2. is there a possibility to make pebble store data on S3? Others looks good to me. Migrate from unnecessary cgo to pure go is a big plus to me

[caesarjuly]

1. is pebble fully compatible with rocksdb? which is saying we can upgrade without change any data format?

2. is there a possibility to make pebble store data on S3?
   Others looks good to me. Migrate from unnecessary cgo to pure go is a big plus to me

All are good questions. I still need some time to figure it out. Let me come back later.

[jiaoew1991]

Hi @xiaofan-luan , I'm looking into the previous PR. It's quite a huge PR. Since I'm still getting familiar with Milvus code, I'd like to know if it's better for me to replace the modules one by one - splitting the huge PR into several small PRs. As @jiaoew1991 shared before, I will keep the RocksDB code and add another implementation based on PebbleDB.

1. Add PebbleDB to the kv module and write basic tests
2. Add PebbleDB to mq module with the required tests
3. Add PebbleDB to configs and other modules
4. Do an integration test and verify the performance
5. Remove RocksDB from the dependency

Any suggestion is welcome :).

Feel free to continue or redevelop a new set based on my PR. I hope this work can be completed, as it will greatly help with the Milvus standalone mode.

[caesarjuly]

Hi @xiaofan-luan , regarding the questions:

1. There are some features not supported by Pebble, according to the readme. But all the features we are using are included, like simply load/set/prefix_scanning, these kinds of primitive APIs. If there is any other concern, please point out.
2. They don't have plan to support s3, according to this issue.

Hi @jiaoew1991 , I created the first PR and left a few questions. Please have a look at your convenience. Thank you~