Figure out how to build and package soroban enabled core builds (Rust)

[MonsieurNicolas]

We're taking a dependency on Rust (as of right now 1.63.0) but Ubuntu seems to be lagging on that front: Ubuntu is on 1.59.0 which is 6 months behind in terms of releases according to https://github.com/rust-lang/rust/blob/master/RELEASES.md

We need to decide how we want to build packages: right now packages make the assumption that they can be build on "stock". Either we allow using "latest" rustc (that needs to be pulled from somewhere), or we'll have to enforce somehow that we're only using the version or rustc available on stock Ubuntu.

[leighmcculloch]

or we'll have to enforce somehow that we're only using the version or rustc available on stock Ubuntu

Developing on 6 month old Rust would be very undesirable. We'd be unable to use new features that have surprisingly been very helpful in the SDK and XDR libs (e.g. const generics). It's not that bad I guess. In the Go ecosystem we always lag 6 months because we support the last 2 versions of Go.

Aside from that it would introduce an inconsistency. Those of us running Ubuntu would be running Ubuntu Rust, which I presume is Rust 1.59 with Ubuntu patches (I assume probably back ported security patches and bug fixes). Those of us running non-Ubuntu would have to choose between running non-Ubuntu Rust 1.59 which would probably not have security patches and bug fixes (because Rust doesn't typically backport them), or running non-Ubuntu Rust 1.63 (or latest) which would mean it'd appear like I have access to features that I don't.

[leighmcculloch]

Another concern: It would significantly increase the risk of when it comes time to do upgrades to a newer Rust, because we'd be upgrading 6 months of changes to Rust in one go, rather than incrementally over time. The choice of when to upgrade would likely be coupled with also updating from Ubuntu. It moves our upgrade model to that more like the waterfall software development flow, where we bundle a large amount of work together and hope for the best.

[MonsieurNicolas]

Agree. I think that because rustc links with dependencies statically, if we were to install latest it would only impact the build environment: it should still be possible to install packages on stock Ubuntu.

[leighmcculloch]

@MonsieurNicolas I think that's correct. According to folks in Rust Discord, the only thing that matters is what version of glibc it is compiled against. See the brief conversation here: https://discord.com/channels/442252698964721669/443150878111694848/1007755179652816936

[graydon]

Re missing features: Const generics arrived in rust 1.51. The rust in ubuntu repos is currently 1.59.

We're currently targeting 1.63 with soroban.

Rust's release notes are here: https://github.com/rust-lang/rust/releases

I'm not seeing anything I'd consider an essential feature for soroban in the delta between 1.59 and 1.63, though I'd be curious to hear if something stands out / if you remember something.

Re inconsistency: the set of debian patches is https://salsa.debian.org/rust-team/rust/-/tree/debian/sid/debian/patches and seems mostly just harmless workarounds for differences between the rust project's canonical build environment and debian's; nothing that's likely to cause a visible difference in our development experience vs. using upstream binaries.

Re large upgrades: debian and ubuntu backport new rust releases periodically (I'm not sure why they've stalled out at 1.59 for the past while); they do not as far as I know have a policy of batching in 6 month units or anything. See the ubuntu changelog http://changelogs.ubuntu.com/changelogs/pool/universe/r/rustc/rustc_1.59.0+dfsg1~ubuntu1~llvm-1~ubuntu1~20.04.2/changelog or the debian upstream repo https://salsa.debian.org/rust-team/rust/activity -- they already have 1.61 queued to land on experimental, it'll work its way through the ecosystem in time.

There are a few reasons to hold back to older rusts in my mind:

1. simpler setup. apt raises less concerns among admins than curl | sh. there's a decent contingent of people who don't like bypassing system packages, and it's yet another thing for people building packages and doing CI tasks to tangle with.
2. less build breakage. when a new rust feature stabilizes it's guaranteed to not break in new compilers, but the converse is not true: new stuff won't work on old compilers. so we're setting a minimum that everyone has to accommodate, and will observe a failure-to-build if they don't keep up.
3. more tool interop. often people build "rust adjacent" tools -- IDEs, static checkers, etc. -- on specific versions of rust and don't keep them quite bleeding-edge.

None of these is a killer issue, but I would prefer if possible to dial it back a bit and retain ubuntu package compatibility. I'm curious how adamant @leighmcculloch feels.

[leighmcculloch]

1. simpler setup. apt raises less concerns among admins than curl | sh. there's a decent contingent of people who don't like bypassing system packages, and it's yet another thing for people building packages and doing CI tasks to tangle with.,

People who use the stellar-core deb file we release shouldn't need to install Rust, so this seems unrelated to how we package stellar-core based on the comments in https://github.com/stellar/packages/issues/168#issuecomment-1213574656 and https://github.com/stellar/packages/issues/168#issuecomment-1213587680. Is that correct?

If this point is regarding developers, developers will probably need rustup because they'll probably need nightly for building deployable small contracts, so I don't think the apt install process is a relevant constraint for developers given that it won't help them install nightly.

Regarding CI, most CI systems come with rustup installed, or provide their own custom methods for versioning rust, and so this doesn't seem like a relevant constraint there either? E.g. https://doc.rust-lang.org/cargo/guide/continuous-integration.html Anyone using more complex CI setups like Jenkins is going to have the chops to install a non-apt Rust.

Would this constraint apply only to the env crate, or also the SDK, and other tooling we build? One ecosystems (apt) slow adoption process is not ideal to constrain on, because it is ultimately just one ecosystem.

2. less build breakage...so we're setting a minimum that everyone has to accommodate

Agreed, but this is a small inconvenience I think. And given that Rust only releases security patches to the most recent stable release it seems highly motivating to stay close to stable's latest so that when security patches are released we are making small jumps not big jumps. I think we will benefit from adopting new features as they're released and it would be unfortunate if we needed to wait 6 months for new features based on external parties adoption patterns.

3. more tool interop.

I'm not aware of this problem in recent times. How often does this happen?

[leighmcculloch]

2. less build breakage. when a new rust feature stabilizes it's guaranteed to not break in new compilers, but the converse is not true: new stuff won't work on old compilers. so we're setting a minimum that everyone has to accommodate, and will observe a failure-to-build if they don't keep up.

I've seen the opposite as well. For example rust-analyzer auto-updates itself (in VSCode) and only officially supports the most recent stable release of Rust (ref). Out of all the tools we use the most important are probably:

• rust-analyzer
• cargo-hack

[graydon]

All fair counterarguments; I'm going to try getting it working with rustup and latest-stable. Will report back here.

[graydon]

See https://github.com/stellar/packages/pull/170 and https://github.com/stellar/stellar-core/pull/3527 for a sketch of this. If they're agreeable and we can merge them we should be able to build a soroban-enabled core package.

[graydon]

Sigh, spoke too soon -- those packages built but of course don't actually have any rust code active in them. My fault, typo. Trying again.

[graydon]

Ok, looks like 14th time (build 1057) is the charm:

packages:

stellar-core_19.3.1-1057.065c02771.focal~soroban1_amd64.deb
stellar-core_19.3.1-1057.065c02771.bionic~soroban1_amd64.deb

docker image:

stellar-core:19.3.1-1057.065c02771.focal-soroban1 

confirming it has rust stuff in it:

$ docker run -it stellar-core:19.3.1-1057.065c02771.focal-soroban1 version
stellar-core 19.3.1.soroban1 (065c027718ba1008448838cf5a6c9fa3d177c9f6)
rust version: rustc 1.63.0 (4b91a6ea7 2022-08-08)