search

rajasekarv / vega

A new arguably faster implementation of Apache Spark from scratch in Rust
Apache License 2.0
2.23k stars 206 forks source link

Why Capnp ? #36

Closed Nimpruda closed 4 years ago

Nimpruda commented 4 years ago

Hi, I was wondering why do you use capnp instead of rust library that would be easier to install ? I feel like the benefits dont overcome the installation overhead.

rajasekarv commented 4 years ago

Though currently not used much, there will be a lot of coordination happening between driver and executor and for that having a good, fast interchange format and RPC library is necessary. At the time when I started writing this library, grpc and protobuf were still not officially supported in Rust. However there are some excellent implementations. I am waiting for one of them to be universally adopted. Tarpc is also a good choice and having no extra language is added bonus. It was also undergoing changes due to async/await stabilization. Only capnproto is matured and stable as of this moment. That is the only reason to use it. Having said this, I am working on stabilizing distributed mode and therefore am thinking of moving to something else.