madsim-rs / madsim

Magical Deterministic Simulator for distributed systems in Rust.
Apache License 2.0
622 stars 41 forks source link

How to add madsim support for opendal? #122

Open Xuanwo opened 1 year ago

Xuanwo commented 1 year ago

Hello, I am the maintainer of OpenDAL. This project is interesting and I would like to extend it with OpenDAL madsim support. Is there any way to do this without copying all opendal code to this repo?

wangrunji0408 commented 1 year ago

Hi! Thank you for your interest in this project. 😄

To build a simulator of your service on madsim, you don't need to copy all existing code. Instead, you will be going to create a mock of all public APIs on top of the madsim infra. Take OpenDAL as an example. I just had a quick look of the opendal docs. It says that it is an abstraction layer over all storage services. So you only need to build a simulator that matchs the semantic of the abstraction layer (Operator and Object), and don't have to build for each service implementations. The simulator will consist of a client side and a server side. The server should emulate your service in a single-thread manner (etcd example), and it should be exposed using madsim's network API (etcd example), so that we can access it later from the client side (etcd example). All of them could be put in your crate with the #[cfg(madsim)] condition, so that when the cfg is not enabled, this crate still behaves the same as original.

For more details, you can refer to our simulation code of etcd-client, rdkafka, and aws-sdk-s3. You are also welcome to direct message me later. Looking forward to seeing madsim-opendal soon. 😜

Xuanwo commented 1 year ago

This issue, along with https://github.com/apache/incubator-opendal/issues/1871, provides a great example of how open source works.

Win, win, win!

Xuanwo commented 1 year ago

This task has been started in https://github.com/apache/incubator-opendal/pull/2006