My understanding is that all the file and database accesses in neptune-core are now async friendly, with the exception of the wallet and mutator_set that use twenty_first::storage.
The primary obstacle here is that twenty-first does not presently have any tokio dep. Everything in the crate is sync/blocking.
afaik, twenty_first::storage is the only module in twenty-first that performs I/O. That makes it different from the rest of twenty-first.
The options I see are:
Add tokio dep to twenty-first.
move twenty-first::storage into neptune-core.
move twenty-first::storage into its own crate.
[1] would probably be the easiest/quickest. But it feels kind of wrong to have a crate with mixed sync/async public API.
[2] and [3] have the obstacle that twenty_first::util_types::mmr makes use of twenty_first::storage. So it might need to be moved into neptune-core as well. but a question arises, does tasm-lib or triton-vm depend on mmr?
Another consideration is that we have twenty_first::sync and neptune_core::util_types::sync::tokio with types that are sync/async mirrors of eachother, but they are split into different crates.
And also we have a sync levelDB wrapper in twenty-first::storage/leveldb and an async wrapper in neptune_core::database::neptune_leveldb.
Moving the twenty-first types into neptune-core would enable putting these types together in an organized fashion, eg neptune_core::database::leveldb::{sync, async} and neptune-core::locks::{sync, async}.
The same could also be done in a new crate, but the leveldb/storage stuff is so different from the locking stuff, really they should each be in their own crate. But I don't see enough value in doing that.
So I think that [2] is the cleanest and best path forward, that cleans up some dirtyness.
I will move forward with prototyping, to see if there are any unforseen problems.
dan-da
commented
3 months ago
My understanding is that all the file and database accesses in neptune-core are now async friendly, with the exception of the wallet and mutator_set that use
twenty_first::storage.The primary obstacle here is that twenty-first does not presently have any tokio dep. Everything in the crate is sync/blocking.
afaik,
twenty_first::storageis the only module intwenty-firstthat performs I/O. That makes it different from the rest of twenty-first.The options I see are:
twenty-first::storageinto neptune-core.twenty-first::storageinto its own crate.[1] would probably be the easiest/quickest. But it feels kind of wrong to have a crate with mixed sync/async public API.
[2] and [3] have the obstacle that
twenty_first::util_types::mmrmakes use oftwenty_first::storage. So it might need to be moved into neptune-core as well. but a question arises, does tasm-lib or triton-vm depend onmmr?Another consideration is that we have
twenty_first::syncandneptune_core::util_types::sync::tokiowith types that are sync/async mirrors of eachother, but they are split into different crates.And also we have a sync levelDB wrapper in
twenty-first::storage/leveldband an async wrapper inneptune_core::database::neptune_leveldb.Moving the twenty-first types into neptune-core would enable putting these types together in an organized fashion, eg
neptune_core::database::leveldb::{sync, async}andneptune-core::locks::{sync, async}.The same could also be done in a new crate, but the leveldb/storage stuff is so different from the locking stuff, really they should each be in their own crate. But I don't see enough value in doing that.
So I think that [2] is the cleanest and best path forward, that cleans up some dirtyness.
I will move forward with prototyping, to see if there are any unforseen problems.