[Epic] Rust in Defra

[AndrewSisley]

A ticket to note down any notable thoughts/findings when thinking about migrating from Golang to Rust, as I've had this thought a few times and am duplicating effort by not writing stuff down and then re-researching/thinking it.

Notes:

• John has mentioned maybe implementing defra in multiple languages a couple of times .- I am somewhat skeptical of the the cost-benefit of this .- Would need to ensure very strong inter-op guarantees RE persisted data (a very good thing in itself IMO)
• cgo (Rust from Go) comes with a host of problems (nice blog here https://dave.cheney.net/2016/01/18/cgo-is-not-go) .- notable performance cost (use as sparingly as possible) .- large cross-compilation issues .- significantly more complex build
• Shouldn't be too difficult to preserve integration tests in Go (using cgo) in the short-medium term .- Will likely cause significant hassle if we expand our build system to cover more build targets though
• Go from Rust is very difficult and poorly documented .- unless maybe we compile rust to wasm (Golang's wasm support isnt great either, but still likely to be much easier)
• No significant rust GQL libraries, would probably have to make our own (easy for most stuff IMO, harder probably for subscriptions?)
• Not yet looked into ipfs/datastore Rust equivalents, could be a significant loss
• Notable gains .- Massive boost in terms of build targets/options (wasm, no-std-lib, features, etc) .- Rust has far better language inter-op, should make it much much easier to write defra wrappers in other langs .- Rust (rust-docs, safer lang, no GC, etc)
• Other notable losses/costs .- Dev-time to migrate .- Significant code complexity gain when blending langs (and to build system) .- Integration test risks

  
  ### Tasks

• [ ] https://github.com/sourcenetwork/defradb/issues/2508

[orpheuslummis]

Can you please clarify what you mean by "Rust from Go" and "Go from Rust"?

[fredcarle]

Go from Rust is very difficult and poorly documented

and

Rust has far better language inter-op, should make it much much easier to write defra wrappers in other langs.

seem contradictory. Am I getting this wrong?

[AndrewSisley]

Can you please clarify what you mean by "Rust from Go" and "Go from Rust"?

Calling Go from Rust, and calling Rust from Go

... seem contradictory. Am I getting this wrong?

It is only contradictory if you ignore the many other languages out there, calling Rust from most languages is far easier than calling Go (and calling out from Rust)

EDIT: To expand, I consider Golang's inter-op roughly on par with the JVM (totally crap), whereas Rust is just a bit behind C's (about as good as it gets), with most other compiled langs falling between the two (never tried calling into an interpreted lang, so I can't comment on those)

[orpheuslummis]

Any ideas for a minimal POC, a first step of Rust integration?

[AndrewSisley]

Any ideas for a minimal POC, a first step of Rust integration?

You'd really want to minimize the number of calls between the two langs, so a minimal POC could still be quite large. Would probably introduce Rust at the bottom and only have Go=>Rust calls (as that is easier and makes more sense IMO), but then it takes ages to get the build/inter-op benefits, and you'll likely hit the difficult issues RE using datastores etc.

The gql generate package might be the easiest, but would have the least gains too (dont really care about performance there, or interop atm).

Would be quite a tricky thing to do, but it is very nice that it is getting my head somewhat more focused on the interop side of things