`--artifact-dir` (nee `--out-dir`) Tracking Issue

ehuss commented 5 years ago

Note: this was renamed from --out-dir to --artifact-dir in #13809

Original issue: #4875 Implementation PR: #5203 Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dir Issues: https://github.com/rust-lang/cargo/labels/Z-out-dir

Summary Adds --artifact-dir=PATH flag to cargo build to specify a directory to place final artifacts.

Tasks:

[ ] #5203
[x] #13809
[ ] Remove --out-dir

Unresolved questions

[ ] Multi-target builds (--target <one> --target <two>)
[ ] #7493
[ ] #6100 — concern about similarity with --target-dir, proliferation of flags
[ ] #6313 — concern about name collisions in workspaces or with examples
[ ] What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?
[x] Confusion with rustc's --out-dir (#13809)
[x] Discrepancy or confusion with build.rss OUT_DIR (depends on if build.rs is generating intermediate or final artifacts) (#13809)

ultrasaurus commented 4 years ago

I find this feature very helpful, since I get output in a directory named like this ./target/debug/build/mylib-79d5eadf36eadf53 and with a build.rs file there are two similarly named directories.

However, it doesn't seem that OUT_DIR environment variable is set to the value of --out-dir when build.rs is executed. Is that intentional? is there some other way to discover the value of --out-dir?

haze commented 4 years ago

+1 on this issue. I have a program that uses a template directory for rust compilation and would like to now have to recompile everything. Yes I could symlink all the folders within target, but it would be nicer to be able to specify the binary output path. I'm not sure why this is such an overlooked issue.

robinbudd commented 4 years ago

When will this feature be available in the stable release? It would greatly simplify my build/CI pipeline.

Chris--B commented 4 years ago

+1 on this. This greatly improves the workflow on our CI, and we'd like to avoid using a nightly compiler just for this.

What needs to happen for this to move forward?

lf- commented 4 years ago

This issue makes nmattia/naersk unusable on stable rust. (another build system issue)

golddranks commented 4 years ago

This would streamline some of my Dockerfiles and build scripts quite nicely.

Diggsey commented 3 years ago

Is there a reason this doesn't work with cargo rustc?

benesch commented 3 years ago

I just published a crate called cargo-out-dir to crates.io, which can ease some of the frustration while we wait for this to stabilize. All it does is print out the current' crate's out directory, as determined by running cargo check --metadata-format=json:

$ cargo install cargo-out-dir
$ cargo out-dir
/home/benesch/path/to/crate/target/debug/build/foo-9d9f15d3d084bc3a/out

simonbuchan commented 3 years ago

Another potential use case for out-dir is making cargo run work nicely with building MacOS apps, which expect a structure like:

My App.app/Contents/
  Info.plist
  MacOS/actual-binary
  Resources/foo.png

in order for native APIs like (the moral equivalent of) [NSImage imageNamed: @"foo"] to work.

But in this case it would be more helpful to be able to put out-dir = "dist/My App.app/Contents/MacOS" in Cargo.toml somewhere, presumably in the [bin] section. Not sure if that makes anyone more or less happy?

workingjubilee commented 3 years ago

--out-dir should not have this name because --out-dir on rustc performs almost the exact function of --target-dir, and to have this be a copy directory command on cargo instead would be confusing. A name like --copy-dir would suffice, however. I believe one of the proposed solutions for #6100, such as having the ability to independently specify "scratch" and "final" build artifact directories is worth investigating.

piegamesde commented 2 years ago

What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

I run into this regularly (multiple times a year), and I am not the only one. At the moment, people hard-code ./target/release/binary or some variant of it like $CARGO_TARGET_DIR/$TARGET/$APPNAME for copying their binaries in installation scripts. This quickly breaks apart in the following cases:

The cargo target dir has been moved to a custom value
Cross compilation.

Cargo's current file handling is unfortunate, and stabilizing this value would help a lot. Specifically, all temporary build files should be freely movable to an external directory, while the build outputs (i.e. the binaries or shared objects) should land in a fixed path for easy packaging.

(In an ideal world, ./target would only contain what --out-dir currently provides by default, and all other build artifacts would land somewhere user-configurable with a default to $XDG_CACHE_HOME/cargo-target/appname or something)

bugproof commented 2 years ago

@piegamesde exactly, I'm used to having bin and obj nice and clean. There should be an environment variable or make it configurable from file as well (--out-dir )

vabka commented 2 years ago

Why is it still unstable? I need to use some dark magic to build in debug/release profiles in one dockefile

...
# If release is set to any value then PROFILE env var becomes 'release' and then left unchanged
# Otherwise PROFILE env var becomes empy and then changes to debug 
ARG release
ENV PROFILE=${release:+release}
ENV PROFILE=${PROFILE:-debug}

RUN cargo build --frozen --offline --target x86_64-unknown-linux-gnu ${release:+--release}

FROM docker.io/ubuntu:22.04 as prod
COPY --from=build /target/${PROFILE}/executable /
...

dpaoliello commented 2 years ago

What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

The key scenario for this is integrating with other build systems, especially where you are cross-compiling: since Cargo will place the build artifacts into a target-triple directory, you need to teach the outer build system about Rust's target-triples so that you know where to look to find the build artifacts.

I'd mostly be happy if there was an option to suppress the target-triple directory (and the release or debug directory).

workingjubilee commented 2 years ago

The key scenario for this is integrating with other build systems, especially where you are cross-compiling: since Cargo will place the build artifacts into a target-triple directory, you need to teach the outer build system about Rust's target-triples so that you know where to look to find the build artifacts.

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

detly commented 2 years ago

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

Just from experience with various (non-embedded) CI and embedded build systems, it's often easier to determine a path for the binary artifacts early and up-front and then use that everywhere, than try to reverse engineer your build tool's scheme for placement. Especially if there's no agreement that the build tool's placement will remain stable.

Consider also that with CI/CD you're configuring things in YAML, so no functions to call like cargo_path_for_binary(app_name, target_triple, profile) - it's copy and paste everywhere. GNU Make at least has functions, but they're not fun to figure out.

There may be more concrete reasons too, that's just what came to mind.

workingjubilee commented 2 years ago

CI configuration languages generally accept variables, indeed this is virtually required for making a CI matrix work. And in a project I have recently been working on which requires constructing a custom build system, I made it so cargo is always invoked with a --target, even if it is just echoing the host tuple, precisely because I don't like the fact that binary artifacts appear in /target/release or /target/release/blah-blah-blah-blah/ based on whether you invoke cargo with --target or not. But neither does it work well to potentially admix the artifacts of different targets.

piegamesde commented 2 years ago

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

One problematic use case I ran into was with some application that was built with Meson. The problem is that it just copied ./target/release/appname into the installation folder. Now I wanted to add cross compilation support for it, and had to hack my way around so that it would take ./target/target-triple/release/appname instead. I think the situation would be half as bad if we didn't have this weird two-level stuff going on based on whether the project is cross-compiled or not.

dpaoliello commented 2 years ago

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

Just from experience with various (non-embedded) CI and embedded build systems, it's often easier to determine a path for the binary artifacts early and up-front and then use that everywhere, than try to reverse engineer your build tool's scheme for placement. Especially if there's no agreement that the build tool's placement will remain stable.

Consider also that with CI/CD you're configuring things in YAML, so no functions to call like cargo_path_for_binary(app_name, target_triple, profile) - it's copy and paste everywhere. GNU Make at least has functions, but they're not fun to figure out.

You might know the target triple within the Rust project, but if you are wrapping calls to Cargo from another build system then there are likely other non-Rust projects in that build system that are unaware of Rust's target triples. It's much easier in a build system to assume that the built artifact is $OUTPUT/$PROJECT.so than to have to thread the target triple (and knowledge that a subset of projects are Rust) throughout the rest of the project so that you can generate $OUTPUT/target/$TARGET_TRIPLE_IF_RUST/$FLAVOR_IF_RUST/$PROJECT.so.

Things only get worse if a project can customize its target triple: it might be trivial to create some Make/CMake function to generate the target triple based on the target architecture of the build, but if an individual project can customize that triple (e.g., MUSL vs GNU, MSVC vs GNU, or using a none target for no_std) then the external projects need to rely on implementation details of how that project is configured to know where its output is.

hashworks commented 2 years ago

As a practical example, this is a problem when one builds and packages mdcat - besides the binary target/release/mdcat additional files are produced, which one needs to locate in a (IMO) unstable way:

$ git clone https://codeberg.org/flausch/mdcat
$ cargo fetch --locked
$ cargo build --frozen --release --target-dir=target
$ ls target/release/mdcat
target/release/mdcat
$ ls -d target/release/build/mdcat-*
target/release/build/mdcat-2a83e790d2b3f7a6  target/release/build/mdcat-7decb70a642f31fb
$ find target/release/build -type d -path "target/release/build/mdcat-*/out"
target/release/build/mdcat-7decb70a642f31fb
$ ls target/release/build/mdcat-7decb70a642f31fb/out
completions  _mdcat  mdcat.bash  mdcat.fish  _mdcat.ps1

mcandre commented 1 year ago

I need this --out-dir feature in order to automate archiving binary artifacts (e.g. tar -czvf ..., zip -r ...), when I release my Rust apps.

Not a fan of how the flag is teased in non-nightly cargo's help menu, only to bail when invoked. Don't bother me. Either let me use the flag, or at least drop it from the CLI documentation.

Alternatively, I could use nightly, and require my users to use nightly. For DevOps users and other fans of pinned, immutable versions, nightly is not a valid option.

Alternatively, I would have less need for this feature if cargo simply nested binary artifacts in a more archive amenable tree structure, like target/bin, by default. Currently, cargo flattens out binary artifacts--the most important part of the build--polluting recursive archives with sibling junk files.

Alternatively, one could write a custom shell script to accomplish this. But that wouldn't work very well across all Rust projects, and it would needlessly restrict Windows devs, and proper robust shell programming is rather a dark art, and shell scripts substantially depart from the Rust ecosystem.

Alternatively, one could configure archive commands to strip out the junk sibling files, but that's a maintenance nightmare, as cargo will surely (as any package manager reasonably would) introduce additional file patterns in the build tree internal directory over time.

Alternatively, one could configure archive commands to look exclusively for file patterns matching target/(debug|release)/<all the binaries mentioned in Cargo.toml>(\.(exe|js|wasm))*. Another maintenance nightmare.

I don't want to get too involved with low level details of cargo internal build files. I just want more reasonable defaults. Or failing that, landing --out-dir in the next Rust release.

valadaptive commented 1 year ago

It would be really really nice if this feature would be stabilized. I'm trying to integrate Cargo with Meson, whose developers steadfastly refuse to support artifacts being output into any subdirectory, so being able to output the artifacts into any directory would be extremely helpful.

valadaptive commented 1 year ago

I'm trying to figure out what the blockers for stabilizing --out-dir are. So far, I've been able to gather the following:

Some people think calling the argument "out-dir" is ambiguous and confusing (what exactly is going "out" into it?), especially given that the OUT_DIR environment variable already exists and means something completely different.
- If naming is a problem, why not call it --artifact-dir instead? That makes it more clear that we're specifying the directory that the final build artifacts will be placed into.
Some people want to see a much larger refactoring of how Cargo structures all of its directories, and are concerned that --out-dir would end up as a piece of historical cruft once that refactoring happens.
- However, as others have pointed out, changing the existing target directory structure in any way would likely be a breaking change, given how many projects have come to rely on certain files being in certain places.
- In addition, I see this type of "let's wait until we can fix everything" as a form of cookie-licking. In the 4 years since this issue was opened, it seems that no substantial progress has actually been made on this supposed large rethinking, and it's only ever brought up as a reason not to stabilize --out-dir.
Some people claim that access to build artifacts is solved by allowing people to better specify the location of the target/ directory, in particular via https://github.com/rust-lang/rfcs/pull/3371.
- However, this RFC doesn't actually address many of the use cases for --out-dir in the first place. For example, as I mentioned above, Meson will not let you pull build artifacts out of any subdirectory. The files themselves must be in the same directory as your meson.build files, no ifs, ands, or buts.

I'm not familiar with the Cargo governance structure--what's needed to push this feature over the finish line?

poliorcetics commented 1 year ago

Some people claim that access to build artifacts is solved by allowing people to better specify the location of the target/ directory, in particular via https://github.com/rust-lang/rfcs/pull/3371.

As the author of that RFC: it doesn't solve the problem here from a quick read through. Or at least, that's not the intended effect and won't help anymore than setting CARGO_TARGET_DIR in the cargo config.

Meson will not let you pull build artifacts out of any subdirectory. The files themselves must be in the same directory as your meson.build files, no ifs, ands, or buts.

Well, another build system being recalcitrant to change is not a good argument for changing cargo IMO: cargo is made to play nice with itself, maybe some more Rust-specific tooling and not much else

If meson wants not to do something that's on them, not on every other tool on earth to fix it.

Outside of that, being able to specify an artifact directory (I like --artifact-dir) would indeed be nice !

valadaptive commented 1 year ago

Well, another build system being recalcitrant to change is not a good argument for changing cargo IMO: cargo is made to play nice with itself, maybe some more Rust-specific tooling and not much else

It's true that Meson's decision in particular is a bit silly, but I feel like some basic concessions to allow Cargo to integrate into other build systems are necessary.

Either:

Cargo is a build system for multi-language projects
Cargo makes itself usable as a part of build systems for multi-language projects
Cargo doesn't integrate in any way with projects that use languages other than Rust. If you want to incorporate Rust into other projects, use another Rust build system.

Option 1 is explicitly far outside of Cargo's scope for good reasons. Option 3 seems unnecessarily restrictive, and right now (as far as I know) there is no "another Rust build system" that supports external crates. That leaves option 2.

eli-schwartz commented 1 year ago

Option 3 seems unnecessarily restrictive, and right now (as far as I know) there is no "another Rust build system" that supports external crates.

Well, meson has a couple of in-flight changes with the intention of supporting external crates in the near future, and at least two major stakeholders whose choice of build system is "meson or bust" that want to utilize this. I can't promise on a precise timeline but we have been chipping away at the issues involved, pieces have landed, and there are practical PoCs.

It will definitely not suit everyone's taste, in particular the chances are approximately 0.000001% that we will ever support build.rs, but the hope is that those will be small enough in number that people can write a meson.build 'wrap file' for a handful of projects that need it and let meson automatically parse Cargo.toml for the rest.

I suspect we'll be finished before rust adds a way to specify the precise output file locations of its build artifacts.

GrantGryczan commented 9 months ago

I believe this comment addresses most of the concerns against stabilizing this issue (I don't know of any others), and I don't see anyone raising objections to it.

This issue is still labeled as "waiting on feedback", so what further feedback is needed? This flag would simplify my Docker setup, so I'd love to see it stabilized. What's left to be done until it can be?

soloturn commented 6 months ago

i do hope the one discussing about this flag will not be dead until this gets enabled in stable :)

workingjubilee commented 6 months ago

Can someone PR a rename of the flag from --out-dir to --artifact-dir as a first step to stabilizing this? It seems there was consensus on that.

soloturn commented 6 months ago

Can someone PR a rename of the flag from --out-dir to --artifact-dir as a first step to stabilizing this? It seems there was consensus on that.

where did you get the impression from that out-dir is not good enough, and what is the purpose to block it even longer?

valadaptive commented 6 months ago

where did you get the impression from that out-dir is not good enough, and what is the purpose to block it even longer?

Confusion with the OUT_DIR environment variable, which is where build.rs scripts should place intermediate outputs.

Can someone PR a rename of the flag from --out-dir to --artifact-dir as a first step to stabilizing this? It seems there was consensus on that.

Created #13809. Doesn't work properly right now, but hopefully it gets the ball rolling.

workingjubilee commented 6 months ago

To add to what was already said, cargo itself already has this inside its implementation:

https://github.com/rust-lang/cargo/blob/eee4ea2f5a5fa1ae184a44675315548ec932a15c/src/cargo/core/compiler/build_config.rs#L39-L44

If the actual code inside cargo does not think out-dir is the best name to use, that seems sufficient to take as a sign, y'know?

epage commented 6 months ago

What would help move this along is if someone could summarize the state of this thread and the feature in general with the goal of catching everyone up on this. Keep in mind, we have over 1300 issues in our backlog and it takes time to come up to speed on any specific thread. That then needs to happen for the whole team to be able to sign off on stabilization. While this is a blocker for some, there issues that are a blocker for others and we have to pick where we put our time. Yes, there was a comment earlier that covered some of the high level topics but was more focused on addressing them, rather than summarizing the conversation to help others come up to speed.

As for use cases, I was recently thinking about this as I was a way to unblock moving the target dir into a central location after rust-lang/rfcs#3371 is stabilized. The vague thought I had was that we could have target-dir and artifact-dir (forgetting about this thread, I also was considering that name change) config fields and target-dirs default changes to a central location while we default artifact-dir to where final artifacts currently go. artifact-dir could support variable substitution as proposed in rust-lang/rfcs#3371. for target-dir. I've not looked into the details for any incompatibilities in this (this is a pretty far out there idea).

epage commented 6 months ago

wrt naming to move forward #6790 without blocking that on any of what I just said, I was wondering if this is a popular enough of an unstable feature that we should offer a transition route for users by supporting both and warnings users that the old one is going to be removed.

There is also the name itself. We use export_dir internally but I think that makes sense for inside of the code for what is happening while, depending on how we design it, artifact-dir better describes what the end-users are trying to control, helping to avoid confusion over what gets included (ie no tests iirc)

valadaptive commented 6 months ago

What would help move this along is if someone could summarize the state of this thread and the feature in general with the goal of catching everyone up on this.

I think a lot of the discussion has been happening in #6100. As far as I can tell, it mostly consists of many Cargo users saying "please stabilize this flag" interspersed with some others wanting to hold off on stabilization until Cargo's entire output directory structure has been redesigned:

Sep 2018 from withoutboats:

We discussed this & also the overarching issue a bit in the cargo meeting yesterday. The sort of overarching viewpoint was that it would be ideal to expose these kinds of knobs through a set of lower level primitive commands on top of which sit the more user oriented commands (git has this distinction for example, calling them "porcelain" and "plumbing").

Apr 2021 from matklad:

Coming back to this after a couple of years, I feel that maybe adding --out-dir is indeed premature, and a brainstorm of "layout of ./target dir" is required before that. I feel there's a bunch of problems with target, and maybe some of them are worth solving in batch.

Jul 2022 from dpaoliello:

Unfortunately changing the layout of the target dir may require the --out-dir feature: currently it is the only officially supported location where the Rust build artifacts can be found, so changing the layout would be a breaking change.

Instead, if we could have an output directory that is guaranteed to only contain the final artifacts, that would enable us to declare the layout of the target directory to be unstable and push folks away from relying on its contents.

An alternative is to move towards deprecating the target directory altogether, and instead introduce the concept of an output directory (or output directory per crate-type) and a temp directory.

The frustrating part for me, and I believe many others as well, is seeing people make these proposals as reasons not to stabilize an existing and wholly implemented feature and then do no work on exploring the design space of those proposals. I don't expect them to champion such work on their own, but some help would be nice if they're arguing against a feature that's already implemented.

I was wondering if this is a popular enough of an unstable feature that we should offer a transition route for users by supporting both and warnings users that the old one is going to be removed.

My implementation of renaming it to --artifact-dir ran into a bootstrapping issue where building Cargo itself (or its dependencies) seems to require the --out-dir flag, so I think this is necessary for that reason alone. Is there an existing mechanism for warning users about deprecated flags? If so, I can work on adding it to my PR.

nils-werner commented 6 months ago

I'm also looking forward to seeing this option merged quickly. In that spirit: Is it really necessary to rename the option?

I understand that the worry is that there could be confusion between --out-dir for cargo and the environment variable OUT_DIR for build.rs, but doesn't it mean exactly the same in both cases?

In both cases you're asking the build tool to place it's output artifact in a certain directory, it just happens that in one case you're asking cargo, and in the other instance someone (cargo) is asking build.rs. But the operation and meaning is exactly the same, no?

valadaptive commented 6 months ago

In both cases you're asking the build tool to place it's output artifact in a certain directory, it just happens that in one case you're asking cargo, and in the other instance someone (cargo) is asking build.rs. But the operation and meaning is exactly the same, no?

No. As I understand it, the OUT_DIR environment variable tells build scripts where to place intermediate artifacts (it's the build directory within the target folder) whereas --out-dir tells cargo where to place final artifacts (binaries and libraries).

workingjubilee commented 6 months ago

Two people have asked "why rename the option?" with an air of "...when we could ship what's currently there, and get it faster?" I wonder if people imagine that, since we have been waiting so long, if someone renames the flag we will have to wait another long time for it to stabilize?

That doesn't happen, in my experience. As the minimum wait from FCP to stable release is 6~12 weeks^0, and the only prior users were necessarily using the nightly toolchain, I have seen teams rename something and then immediately start the stabilization clock.

But I have also seen a team member that does not think something is "finished" decide to raise a blocking concern. Including on naming. And that by definition can delay the stabilization process indefinitely. I speak in generalities of course.

"Should the name be --out-dir? Probably not?" is not one but three of the Unresolved questions on this very tracking issue. Having observed a lot of FCPs, asking for an FCP to start without a good answer for all unresolved questions is a good way to materialize a blocking concern for each unanswered question. That is why features like this one are made closer to their finished, and thus stabilized, forms, by removing a highly contentious and confusable trait.

workingjubilee commented 6 months ago

@valadaptive In https://github.com/rust-lang/cargo/issues/6790#issuecomment-1724295494 you asked this:

I'm not familiar with the Cargo governance structure--what's needed to push this feature over the finish line?

Note: I am not a T-cargo member. But to answer to the best of my ability:

T-cargo uses rfcbot to do FCPs like most of the Rust teams. That means any T-cargo member can call for a T-cargo FCP, and that is what you're looking to convince someone to start. There is T-cargo's unstable feature process that explains everything but... probably skip towards the end, as this has been baking quite a while. More of a background note, but might make things make more sense: like most teams this process is partially templated on the rustc stabilization guide.

There's also interesting tidbits^0 of related documentation in the Cargo contributor guide. Can't see anything about deprecation tags though!

valadaptive commented 6 months ago

https://github.com/rust-lang/cargo/issues/7493 also seems to be a blocker here.

valadaptive commented 6 months ago

Here's a recap of the unresolved questions:

[x] Is --out-dir justified or can we find another way of accomplishing it? #6100 — concern about similarity with --target-dir, proliferation of flags

Over the years, many people in that issue thread have proposed their dissatisfaction with --out-dir and suggested some sort of alternative. As far as I can tell, no design work has actually been put into any of these alternatives. Also, they all seem to require large refactorings of Cargo, and may be breaking changes considering that many build systems already assume the layout of the target directory. As such, I consider them untenable.

[ ] Duplicate artifact tracking issue. #6313 — concern about name collisions in workspaces or with examples

Still relevant and will need to be solved.

[x] What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

See above for all the developers who want this to be added.

An important use case is being able to directly output artifacts where you want them, without having to know where the target directory is or copy out of it as an extra step. Figuring out the former is complicated by the --target-dir flag. This is very helpful when integrating with other build systems, CI pipelines, Docker, etc.

A couple pain points with the current situation seem to be knowing whether to reach into the target/release or target/debug folder, and accessing the folder for the correct target triple when cross-compiling. Passing such information around the build system you're trying to integrate with can be tedious and error-prone.

[x] Confusion with rustc's --out-dir

[x] Discrepancy or confusion with build.rss OUT_DIR (depends on if build.rs is generating intermediate or final artifacts)

Solved by renaming to --artifact-dir (https://github.com/rust-lang/cargo/pull/13809).

As I noted above, https://github.com/rust-lang/cargo/issues/7493 is also a blocker that will need to be resolved before we can stabilize this flag.

epage commented 5 months ago

Over the years, many people in that issue thread have proposed their dissatisfaction with --out-dir and suggested some sort of alternative. As far as I can tell, no design work has actually been put into any of these alternatives. Also, they all seem to require large refactorings of Cargo, and may be breaking changes considering that many build systems already assume the layout of the target directory. As such, I consider them untenable.

That is not sufficient reason to ignore those concerns.

However, since then (besides team members rotating)

We've started talking more concretely on TARGET_DIR layout changes in #5931
rust-lang/rfcs#3371 was created and is moving forward. A potential shiny future for that is that everyone has their unstable-layout TARGET_DIR hidden away with everyone accessing the stable-layout ARTIFACT_DIR

Looking at those, I agree with https://github.com/rust-lang/cargo/issues/6100#issuecomment-1189367488 that having --artifact-dir is likely the way forward for allowing more evolution of intermediate artifacts.

See above for all the developers who want this to be added.

What would be important for organizing all of the use cases is to understand what is in or out of scope (or if we need more, lower level knobs as mentioned).

e.g. https://github.com/rust-lang/cargo/issues/6100#issuecomment-1250332307

Is it possible to make --out-dir option for cargo test also? The use case that I need is to run the binary test file to run under a debugger

A feature may sound like it works for one person or it may indeed work for one person but we need to consider the overall ecosystem and "where will this lead" both in terms of second order effects (not concerned about that here) and what will people find lacking and ask to have changed or extended. We don't have to solve everyone's problem but we need to have a rough idea of what the future is to make sure we are solving the right problems. For me, I'm less concerned with #6100 for this flag but whether we'll need several additional flags after this because we didn't quite solve the right problem.

detly commented 5 months ago

What would be important for organizing all of the use cases is to understand what is in or out of scope (or if we need more, lower level knobs as mentioned).

[...]

A feature may sound like it works for one person or it may indeed work for one person but we need to consider the overall ecosystem and "where will this lead" both in terms of second order effects (not concerned about that here) and what will people find lacking and ask to have changed or extended.

I think that to a large extent, the need being addressed can be split into two parts:

The need to predict where certain build artifacts will be placed.
The need to control where certain build artifacts will be placed.

The "certain build artifacts" can also be split into two parts:

Build artifacts that are needed for actual deployment, perhaps called "final" build artifacts.
Build artifacts that are needed to analyse coverage, cache build stages etc. perhaps called "intermediate" build artifacts.

By way of example, I have recently been trying to implement fine-grained code coverage reporting in CI on a Rust project. It is necessary to tell the coverage tools where both the coverage instrumented binary artifacts are, and where the coverage data is. The latter can be controlled through environment variables, so perhaps we can ignore that. But the former is extremely hard right now.

For a native dev build, they're in target/debug.
For a native test build, they're in target/debug.
For a native release build or any other profile, they're in target/<profile>
For a cross build, they're in target/$RUSTC_TARGET_ARCH/<profile> where Cargo has been invoked with --target=$RUSTC_TARGET_ARCH

So to simply generate the command lines for a coverage tool in CI, I need to implement this logic in CI or a build script. I can't just have CARGO_ARTIFACT_DIR="target/$CARGO_PROFILE", because of the maybe-there maybe-not target component, although @workingjubilee's trick works for that. I cannot have CARGO_ARTIFACT_DIR=target/$RUSTC_TARGET_ARCH/$CARGO_PROFILE, because dev and test profiles map to a debug directory name.

I also can't just use the JSON output of Cargo because there is no part of that that gives me the base directory, only the full path to each individual artifact. The base directory is required eg. for most instrumentation analysis tools, for caching and storing artifacts consistently, and because the full list might actually be too long for many shells' limit on environment variables.

So I need to implement a more complex mapping that duplicates Cargo's internal artifact directory construction logic. In shell script. Or the CI config syntax. Or another build script language. This is undesirable because:

it's non-trivial and error prone
(AFAIK) it's not an API detail of Cargo, and could change, so it's fragile
it's making everyone who needs this do the work over and over instead of doing it in one place ie. Cargo

This is not just an issue for code coverage. This applies to:

integrating Rust projects into more complex build configurations eg. OpenWRT, Meson projects, installer build systems, almost any mixed-language project really
trying to manage caching in CI systems to reduce test turnaround time
trying to pass artifacts between CI jobs

Now consider that if I were just able to predict (or extract from Cargo's output) where artifacts will end up, I can effectively meet "the need to control where certain build artifacts will be placed" by writing cp "$PREDICTED_ARTIFACT_DIR" "$DESIRED_ARTIFACT_DIR". Hell, someone could write a tool cargo-with-outdir that does nothing but force certain flags/parse certain output and do the copy for you. There may be other cases where it is simply necessary to dictate this directory and not predict it, I don't know.

I'll also preempt the argument that there are tools that abstract this away for some of the use cases I mention eg. cargo-llvm-cov for coverage, cargo-chef for CI caching, cargo-nextest for test artifact passing, OpenWRT's own Rust build integration. That's a bit of an own-goal, because it's actually just listing places where the internal logic of Cargo's directory naming has to be duplicated, or the logic of capturing Cargo's output just to collect artifacts in a common directory has to be duplicated. Those tools do not cover every use case eg. to use exclusion filters for Rust code coverage, I need to ditch cargo-llvm-cov and use eg. grcov or gcovr, which... need telling where the artifacts are. When cargo-nextest's intermediate archive needs winnowing out to be copied to various CI jobs with different responsibilities... I need to know where the artifacts are. Not every niche is going to be covered by a 3rd party tool, and not every 3rd party tool is going to have a common subset of Cargo-output-path-predicting code that can be extracted into a 4th party tool.

In the last four years of using Rust in earnest and trying to do novel things related to code quality, instrumentation and testing, I just keep running up against this. I don't mean this as a kind of pressure to just "get it done", because I respect (a) Cargo's commitment to quality of API and (b) the fact that (veeery much like the --deps-only issue) a lot of different goals have kind of been simplified into "give us --some-flag!", and it might not actually work in a way that achieves those goals. But it is to say that there is probably a very common pain point here between users and use cases, and hopefully the above articulates that a bit more, and it really does seem like --artifact-dir or something like it must be pretty close to addressing those.

soloturn commented 3 months ago

rust and cargo has so many flags and environment variables, which i find quite confusing. there are legacy tools which have a logic variable naming, like cmake: https://cmake.org/cmake/help/v3.0/manual/cmake-variables.7.html, it has a couple of directories, but for profiles it operates with postfixes. for building, the default cmake is very simple. either change directory and build, or call cmake -Bbuilddir -Ssource. there is no cache, no artifact, no out, no target, no lib etc etc. if the file is there it uses it, if not it produces it.

if you want to have a more modern example, bazel has the following: https://bazel.build/remote/output-directories. it is designed for buils on a cluster, and extreme reuse of build artifacts.

therefor i woudl support a design ... if the designer would be prepared to remove 90% of the existing varialbes would be already a huge advancement :) my personal opinion of course - just coming from my current frustration of hours of build time of COSMIC desktop because it recompiles the same libraries again and again.

epage commented 3 months ago

So I need to catch up on some things earlier in the thread but needed to record some thoughts real quick.

Someone brought up multi-target builds (cargo build --target <one> --target <two>). If could error if we build for multiple targets and the artifact-dir is the same value. This could be better controlled with templating in the artifact-dir.

While I don't dig into the blockers for this, https://github.com/rust-lang/cargo/issues/14125#issuecomment-2258708917 explores a potential cohesive picture of what we should be doing

rust-lang / cargo

`--artifact-dir` (nee `--out-dir`) Tracking Issue #6790