Prepare for additional Rust envs and prevent forked rust versions from holding back updates

[mattiekat]

I recently worked on updating Rust #2619 which was the hardest it has ever been because now with the forked version of the rust compiler that solana uses, I had to track down multiple packages that required a newer version of rust to be built with. I think it is inevitable based on how much work it took us to resolve conflicts when we added solana to the monorepo that as we add additional rust envs it will only get more and more difficult to sort out these conflicts.

Rust does support multiple package versions, but only if it is either 1) 0.x. and 0.y. where x != y 2) x.. and y.. where x != y

So what can (and has happened) is we end up patching solana to use a newer library version.

But there is another problem, there is also the rust language version, which sometimes packages set a min rust version to be compiled with, and it is not something that is checked because (fun fact) you can't set a max version. Rust assumes you will always be updating because rust is very good at maintaining backwards compatibility. See also https://doc.rust-lang.org/cargo/reference/manifest.html#the-rust-version-field

When we add the next env that uses rust, it is very possible we will not be able to resolve the 2 or 3 way dependency conflicts without a lot of forking and patching which we don't want to maintain.

The good news is I have a solution which should minimize how many conflicts we run into:

• Split solana programs split into a program-lib and a program.
  • The program must not be built by our workspace and is only built by solana rust (or the env's version of rust of your choice)
  • The program-lib is built by both.
  • This means we can have different crate versions if we have to and use feature flags on the rare occasions we have to in the program-lib to sort out conflicts.
• hyperlane-core is either split into hyperlane-types and hyperlane-core OR (my preference) we use more feature flags to create a truly minimal version of it that can be built by everything; the main goal for hyperlane-core is avoid deps and reduce conflict risk.

It means there would be multiple workspaces technically, but there would be 1 main workspace, and then workspaces that only build smart contracts.

The only alternative I can see to this is some sort of rlib-linking process once things are built, but that feels much more prone to breaking and is very non-idomatic as it would prevent other people from being able to import our crates into their rust projects and probably require unsafe code.

[daniel-savu]

Feature flags can get hard to reason about when the sections they cover get too big. So in principle I'd be ok with having a hyperlane-types crate. However the per-environment workspace approach seems more maintainable, even if it sounds like it would increase the build size / time.

[nambrot]

@tkporter i assume we keep this open?