Async API on hypercore

[bltavares]

Feature Request

Summary

The design of hypercore on node is quite callback oriented, and operations may happen at some point. We could provide an async API that feels more like Rust instead of callbacks.

Motivation

Translating the callback oriented operations from dat.js to Rust seems to prompt a more asynchronous API. Now with the stable async/await support, it seems to be ideal to provide such API as we would be dealing with file operations and network.

This would be a breaking change as I don't know yet how to propose maintaining both the sync and async API on the same crate.

Guide-level explanation

We could have an API such as:

let mut feed = hypercore::open("./feed.db").await?;

feed.append(b"hello").await?;
feed.append(b"world").await?;

assert_eq!(feed.get(0).await?, Some(b"hello".to_vec()));
assert_eq!(feed.get(1).await?, Some(b"world".to_vec()));

This would require to change the underlying storage to benefit from async operations as well.

Reference-level explanation

Async traits are not yet stable, but we could use https://crates.io/crates/async-trait to implement it.

We could implement on https://github.com/datrs/random-access-storage a blanket implementation:

#[async_trait]
trait AsyncRandomAccess {
  async fn async_open(&mut self) -> Result<(), Self::Error>;
}

impl AsyncRandomAccess for RandomAccess {
 async fn async_open(&mut self) -> Result<(), Self::Error> {
 async_std::task::spawn_blocking({ self.open() }).await
}

}

This is how far I've thought about it and have a concrete example. I would need to test it out further, but I would like to open for feedback.

Drawbacks

• How to maintain both a async and sync API?
• We would require all consumers to also use an async executor if we don't expose both APIs.
• Async trait is not stable yet, requires an extra dependency on the stack

Rationale and alternatives

Not doing this would mean that networking might not be as performant as it could, as access to file would block the operations.

Unresolved Questions

Many:

• How to maintain both a async and sync API?
  • Should we keep only an async API, like node?
  • Is it ok to require an async executor?
• Is it ok to add async_trait as a dependency?
• Async trait is not stable yet, requires an extra dependency on the stack

[yoshuawuyts]

How to maintain both a async and sync API?

I would recommend only maintaining an async API, and making the switch throughout.

We would require all consumers to also use an async executor if we don't expose both APIs.

That's accurate; within Rust itself that should be no problem (when using async-std). But for integration with other languages I'm not necessarily sure how that would work.

Is it ok to add async_trait as a dependency?

I'd say so; the only downside of this is some extra boxing, compile times, and weird type signatures in docs. But overall it makes life significantly better, so I'd def recommend using it.

[bltavares]

@yoshuawuyts thanks for the inputs. Would you also migrate random-access-storage completely to be async, or would you suggest to use the blanket implementation route?

[yoshuawuyts]

I'd probably migrate it as well.

[bltavares]

There is now a draft PR using async/await on APIs that hit storage: https://github.com/datrs/hypercore/pull/103

It's built on top of many other PRs tho, so it might take a while to be reviewed and land.

[bltavares]

@Frando Are you ok if we land PR to close some PRs?

We could release it as a RC version, and them try to investigate blocking or smol to abstract over the async executors in future releases.

[bltavares]

Landed on the beta versions