search

rust-lang / crates-io-cargo-teams

the home of the crates io team
5 stars 11 forks source link

Cargo pre-planning: architecture #11

Open nrc opened 5 years ago

nrc commented 5 years ago

cc @Xanewok, @nrc, @dwijnand , @alexcrichton , @joshtriplett

nrc commented 5 years ago
nrc commented 5 years ago

If we were to factor out some 'plumbing' commands, then I think that would provide an easy way to do custom profiles and dep-only builds. The question is how urgent those things are and if we can go far enough down the plumbing/porcelain route in a reasonable time.

Rustup integration also ties into #12 because downloading binaries is a key part of rustup missing from Cargo.

@Xanewok could you provide some context on build system integration please?

Xanewok commented 5 years ago

Sure. I was mainly investigating integration with Buck build system but similar rules apply to Bazel, GN etc. I did try to write some notes while doing so - they're over at dev-tools-team/rls. (not directly related to RLS - it somewhat helps, but mainly I just needed a place to keep them)

There seems to be two sides to this, each bringing different benefits - integrating Cargo into a a bigger, outer routine and the other way around.

It seems like integrating Cargo into external systems is what more people are interested in. Generally, people want to keep using their own build system, have a clearly defined way how to build Rust on a larger scale and have a convenient access to the Rust ecosystem (crates.io).

Right now, there are a few hurdles on the way to achieve that.

  1. One of the things are transparent inputs/outputs. Almost all build systems are designed around rules/units that have a clearly defined input set and output to achieve minimality and correctness. Currently, the only way to get the inputs is to cargo build with appropriate parameters and snoop around the target directory for rustc-emitted Makefile-compatible .d files and parse them yourself; to get the outputs, you need to use the currently unstable cargo build --build-plan. I did some preliminary work to also extract the unit inputs in the build plan but due to how the compilation process works now, it may be necessary to actually run the build if it hasn't been done already.
  2. build.rs. It can run arbitrary logic, read the untracked inputs (environment, files), create arbitrary files (also outside OUT_DIR), modify existing source files. It's problematic in the sense that it's hard to get a theoretically reproducible build rule for that and it doesn't mesh well with the input/output tracking. Right now other build systems provide an escape hatch rule (no. 3) but they also require to explicitly provide outputs for work tracking. Currently, the 3 most common use cases seem to be:
    1. detecting rustc version and enabling various --cfg flags (solved by RFC #2523
    2. compiling/detecting native dependencies
    3. codegen
    4. (it'd be good to at least provide the build script outputs; possibly also focus on native deps and codegen separately so it can be declaratively specified in the manifest, rather than performed mechanically in the script)
  3. More transparent resolution phase (easier vendoring/access to crates.io). The feature resolution is conflated with version constraint resolution and to easily access the Rust ecosystem and get what people need, with precise versions and features, they mostly create a dummy project, list [dependencies] and do a cargo build, reusing the output and/or downloaded packages. I know I'm a bit hand-wavy about this but it seems like a real problem with no clear solution so it's worth having that in mind.

Apart from that, there are some more discrepancies that make the Build System <-> Cargo rule translations harder (It's fine that Cargo is opinionated but maybe we could support some of those, like dependencies per crate target?)

The integration other way around would benefit using the usual Rust workflow (RLS, Clippy, Rustfmt), when tools expect a Cargo workflow. One idea might be to formalize a bit an intermediate format (one that Cargo.toml could be desugared to? And which external build systems will have to synthesize from their own rules) that Cargo can understand and possibly build on its own/be reused by dev tools.

dwijnand commented 5 years ago
  • plumbing/porcelain

Interestingly I found that the second commit in the repo outlines porcelain and plumbing commands. 😀

nrc commented 5 years ago

ping @Xanewok, @nrc, @dwijnand , @alexcrichton , @joshtriplett

Some more questions:

alexcrichton commented 5 years ago

Sorry I haven't had a lot of time to dig into this, I was planning on revisiting/posting to this after the 2018 edition is out

alexcrichton commented 5 years ago

Alright I've got some time now! Here's some thoughts I have for this

Custom profiles

Tracked at https://github.com/rust-lang/rust/issues/48683 I think this is a must-have for Cargo by the end of 2019. The end is very near and we practically just need to stabilize it. There's some discussion currently happening on that issue, but all the great groundwork laid by @ehuss I think is overall ready to go modulo a few final points.

Build system integration

I continue to feel uneasy about our progress on this, but I think the best way we can make progress on this is by improving and possibly stabilizing the current build plan. We need to get build scripts and such under control and in the build plan in a way that's easy to integrate with other build systems. I believe that fundamentally build scripts don't pose that large of a problem, and we're just figuring out how to wrangle them.

There's a lot of discussion about this happening on https://github.com/rust-lang/cargo/pull/6213 and I personally think we can get to a point where a build plan can be represented with high fidelity (including build scripts) and quickly generated and all that.

Refactoring/porcelain/plumbing

I don't personally think this is too too pressing for Cargo right now. I can certainly be convinced otherwise though! I think we may want to pursue some degree of this to the end of build system integration, but otherwise I feel like trying to expose the plumbing commands of Cargo is often us "giving up" and not finishing a design, hoping the community can fill in the gaps (which seems to only happen rarely). That's of course not to say that this isn't useful, just that I don't think we should prioritize it just to have it, only prioritize it when it makes sense (like for build system integration)

Rustup integration

I think I'd probably table this for 2019 unless a great need arises. We're definitely getting to the point that rustup is getting to be on its last legs before we have to do something, but I think we're solid enough for now. Across all the various things we'd like to see Cargo do in 2019 I'm not sure how well this fits in and whether it should be prioritized over others.

One thing that we could do to make some progress on this though is greatly improving Cargo's notion of the standard library. Cargo could take over, for example, rustup target add. This is also the whole std-aware Cargo thing, which may be covered by https://github.com/rust-lang/crates-io-cargo-teams/issues/15

Eh2406 commented 5 years ago

Sorry if this is off topic, but you menchand stabilizing the build plan. I think we should have a recommended crate for using the output of build plan and metadata that we guarantee to keep up to date with any changes we make to the output of the commands. I think this will make it easier for out of tree subcommand to say up to date without having to depend on cargo as a library. If the team would be interested in endorsing such a crate I can reach out to the maintainers of the existing explicitly unofficial libraries (cargo_metadata and escargot), to start discussing how best to make something less unofficial.

nrc commented 5 years ago

Discussion from meeting