Open kennykerr opened 3 years ago
And grabbing the target_directory
reported by "cargo metadata" doesn't work. If the crate is coming from crates.io then Cargo builds it in C:\Users\kenny\.cargo\registry\src\<random>\<some-crate>
and the "target_directory" that cargo metadata
reports is C:\Users\kenny\.cargo\registry\src\<random>\<some-crate>\target
- what I need is the target directory for the crate the developer is building directly, not an intermediate target directly only known to the dependency.
I've been recently working on a project where there's need to link the generated binary in user project to another directory and I couldn't find a reliable way to get the target directory either.
Currently, the code just guesses that the target directory is set as default <project dir>/target/
but if the project is located in a workspace or the build.target-dir
is set to something else then this linking simply fails.
Such feature to get the target dir reliably would be greatly appreciated :heart:
The solution: I'd love a CARGO_TARGET_DIR environment variable. There's clearly precedent for this as cargo doc depends on such a thing today.
This has cause and effect backwards. cargo doc
works fine without CARGO_TARGET_DIR set in the environment. Docs.rs uses that as an input to cargo because the source directory is sandboxed, and cargo happens to pass it through to any build scripts it invokes.
Yes, I since noticed that this is an optional input to Cargo:
https://doc.rust-lang.org/cargo/reference/environment-variables.html
Indeed, that's not what I want. I really need the target dir for the most dependent crate being directly built. I've tried numerous unsatisfying hacks. The one that works the most reliably is to read the first path from the PATH
environment variable. This always appears to be:
<target_dir>\<profile>\deps
Although this only works on Windows. And I don't know how "supported" this is but I haven't found a better solution.
Can you provide more information on exactly why you need the target directory? Cargo is designed so that build scripts are intentionally constrained on what they should do, and their only interaction should be through the OUT_DIR.
@ehuss Kenny is out this week but some scenarios are listed here https://github.com/microsoft/windows-rs/issues/979#issuecomment-880271566.
I'll take a stab here at paraphrasing:
The windows
crate has a build script that generates bindings for the developer, sourcing from its built-in metadata or local workspace metadata. Things get tricky when this metadata is delivered via multiple crates in the dependency graph.
For example, envision a developer's crate depending on windows
, windows-fizz
, and windows-buzz
each with their own unique metadata. The build script (macro) running in the context of the developer's crate needs to have access to all the dependent crates' metadata to generate the developer's desired bindings.
One proposed change is to align the normal build script behavior with the cargo doc
build script behavior. CARGO_TARGET_DIR
would then exist and point to the developer's crate target directory and all dependent crates could write their metadata to that path for consumption.
One proposed change is to provide a pointer of sorts to the developer's crate target directory and all dependent crates could write their metadata to that path for consumption. Open to other ideas, of course.
One proposed change is to align the normal build script behavior with the cargo doc build script behavior.
@riverar you're making the same mistake as https://github.com/rust-lang/cargo/issues/9661#issuecomment-877413805, cargo doc
is not related to the target dir. Docs.rs just happens to set CARGO_TARGET_DIR and you've never tried setting it locally.
@jyn514 Ah right, forgot. Thanks!
This doesn't affect the proposed change much. The bottom line is we'd like a pointer to the target directory for the aforementioned reasons.
One use-case I have is to store build-script's own caches. I can use a temporary directory as a scratch space (though it might still be nice to place this somewhere inside ./target
rather than /tmp
), I can use OUT_DIR
for a real output, but I am not sure what's the right place for intermediate artifacts generated by the build script, which are not necessary output, but which I nonetheless want to preserve between invocations.
For example, if I am building some C code, I want to put intermediate .o
files somewhere. Or, my specific use-case, I recursively invoke Cargo to compile some helpers to wasm. I mistakenly tried to use target dir for that, but that broke in all kinds of interesting ways down the line. Looking at how cc
crate just puts .o
into OUT_DIR
, I think it's correct to use that for intermediate artifacts as well, but it isn't obvious from the docs. Will send a PR shortly (https://github.com/rust-lang/cargo/pull/10326).
+1 for this issue as we are distributing dynamic libraries that are hard to build (especially on Windows) internally (by automatically downloading them from a server), and these libs needs to be in the same directory as the final executable.
Without a reliable way to determine TARGET_DIR
(e.g. the user might be cross compiling, or using non-default toolchains), cargo run
will not work out-of-the-box.
FWIW, cxx_build
has the following module to determine target_dir: https://docs.rs/cxx-build/latest/src/cxx_build/lib.rs.html#1-473.
It expects the path stored in the OUT_DIR environment variable as input. I'm not certain it handles all possible situations though.
I'm working on cxx-qt, a code generator build on top of cxx. I need a stable path to output generated C++ headers from build.rs for C++ build systems to be able to find them. I've seen what cxx_build does by walking up from OUT_DIR but that feels quite hacky to me. The least worst idea I've come up with is explicitly setting the CARGO_TARGET_DIR environment variable from the C++ build system so it gets passed to build.rs. I wish I did not have to do this though. Of course, this workaround isn't available if you're not working with an automated system that invokes cargo.
On further thought, for cxx-qt (and cxx), perhaps setting the CARGO_TARGET_DIR environment variable isn't the best idea. I don't want to depend on unstable implementation details of the structure of the target directory. I think it would be better to use an environment variable specific to this purpose. Or better yet, a designated place to put exported build artifacts (#5457).
I think this is quite a different use case from the windows crate. Please correct me if I'm wrong, but I think what @kennykerr and @riverar are asking for is a way to pass data from the build script of a dependency to downstream build scripts. I'm not sure exposing the target directory to build scripts is a great way to do that either. cxx_build uses a public static mut struct together with links
in Cargo.toml for this purpose... which also feels hacky, but I guess it's less hacky than relying on the layout of the target directory? :shrug:
I think it is quite silly that at present it is difficult for a crate to discover where it is currently being built
A use case I have: I want an easy way to run a target (a CLI bin) from an integration test (using the Rust test runner). Not a blocker, just a nice to have.
Some people with vaguely the same use case and some solutions: https://users.rust-lang.org/t/integration-tests-for-binary-crates/21373/9
cargo already makes binaries available for integration tests. cargo_bin!
is one example of this.
Trying to get the binary from just CARGO_TARGET_DIR
has issues because the binary can be dropped in a couple of different places, depending on how you are building the project.
Ah! Thank you! Sorry, I should have looked a bit more .
CARGO_BINEXE
— The absolute path to a binary target’s executable. This is only set when building an integration test or benchmark. This may be used with the env macro to find the executable to run for testing purposes. The is the name of the binary target, exactly as-is. For example, CARGO_BIN_EXE_my-program for a binary named my-program. Binaries are automatically built when the test is built, unless the binary has required features that are not enabled.
https://doc.rust-lang.org/cargo/reference/environment-variables.html (in case it helps anybody who ends up here)
For anyone who might need it, this actually seems to work fine and I've been using it in docify for a while now (gets the crate root of the current proc macro caller when executed within the body of a proc macro):
This seems to do the trick for me:
rustup toolchain install nightly
cargo +nightly config get build.target-dir -Z unstable-options
This variable CARGO_TARGET_DIR
is guaranteed in the cargo documentation, and I'm curious why all the project maintainers refuse to add it. Another words, why don't you remove the description of CARGO_TARGET_DIR
from the documentation?
If you had read it carefully, in the doc, it is under the section of "Environment variables Cargo reads", which is different from the latter section "Environment variables Cargo sets for build scripts". You want your Cargo to sets the variable for your build script, and that is never guaranteed.
From https://github.com/rust-lang/cargo/issues/12673#issuecomment-1721123826:
This implementation could be: if the
CARGO_TARGET_DIR
variable already exists, do nothing, if it does not exist, add it. Is there anything wrong with this?
There is nothing wrong for a feature request like this. But again, if you had looked into issues you linked carefully, there are indeed some open questions needing to address. For example, the concurrent issue for accessing target-dir mentioned here https://github.com/rust-lang/cargo/pull/8710#issuecomment-695047331. Also, we may need to figure out the implication and interaction between this and other features, such as https://github.com/rust-lang/rfcs/pull/3371 and per-user build cache.
This is the workaround I currently use when I need the macro caller crate root in proc macros: https://github.com/sam0x17/docify/blob/main/macros/src/lib.rs#L75-L112
This works in workspaces, outside of workspaces, and regardless of whether we are a crates.io crate or a local crate.
This could probably be adapted to reliably get the proper target directory as well.
Until we get better support for this sort of thing, this is probably the best you can do
Through the information displayed in this image, we can infer the exact path of CARGO_TARGET_DIR
, it's d:\mytoy\target\x86_64-pc-windows-msvc\release
. My problem is solved.
Since OUT_DIR
is unique, then CARGO_TARGET_DIR
is also unique.
Now my question is: why you guys are so stubborn in not providing CARGO_TARGET_DIR
.
Here is my code to find out CARGO_TARGET_DIR
, hope it can help somebody.
fn get_cargo_target_dir() -> Result<std::path::PathBuf, Box<dyn std::error::Error>> {
let out_dir = std::path::PathBuf::from(std::env::var("OUT_DIR")?);
let profile = std::env::var("PROFILE")?;
let mut target_dir = None;
let mut sub_path = out_dir.as_path();
while let Some(parent) = sub_path.parent() {
if parent.ends_with(&profile) {
target_dir = Some(parent);
break;
}
sub_path = parent;
}
let target_dir = target_dir.ok_or("not found")?;
Ok(target_dir.to_path_buf())
}
In addition, I also found a bug in your recent implementation: the two strings d:\mytoy\target\release\deps
and d:\mytoy\target\release
you added to the Path
environment variable are Wrong, it should be d:\mytoy\target\x86_64-pc-windows-msvc\release\deps
and d:\mytoy\target\x86_64-pc-windows-msvc\release
.
I support this feature request. I need this functionality if I want to copy some assets to my target folder. I am using https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html so I can't just hardcode a path.
CARGO_TARGET_DIR
and/or CARGO_BUILD_TARGET_DIR
and/or another name that you define should be an environment variable that is set before build.rs
is called so I can read it just like I am able to read PROFILE
.
fn get_cargo_target_dir()
from @ssrlive is a great workaround. Thank you very much!
I still hope that we can get this feature request with an officially supported env that is less hacky.
Note that directly exposing CARGO_TARGET_DIR
was attempted in the past but was rejected, see https://github.com/rust-lang/cargo/pull/8710#issuecomment-695047331
target/
layout is limited and we are looking at making changes for #5931 + #12633Cheers to weihanglo for finding this context earlier in the thread
So the way to move this forward is for us to look at the use cases and try to identify solutions without CARGO_TARGET_DIR
Doing a quick summary of use cases. If your use case is not listed here, it likely is because it lacked details for me to understand what is happening / why this is needed
CARGO_BUILD_TMP_DIR
--out-dir
so they are next to the cargo-built final artifacts: https://github.com/rust-lang/cargo/issues/9661#issuecomment-1049452886
build script generating C++ header files from the Rust code: https://github.com/rust-lang/cargo/issues/9661#issuecomment-1223301577 The thing we need to understand is if a build script is the appropriate tool for this or if it was the most convenient. Generating C++ headers is not needed for the current package (e.g. "if a package depended on yours through the registry, would it be needed?". This seems like more of a build orchestration concern.
To frame this use case more generally: generate an artifact (anything other than code that gets compiled into the library/executable; for example, C/C++ header, man page, icons...) during a Cargo build which is put somewhere that external tools (for example, a C++ build system, or a package generator) can reliably find them. A build script doesn't have to be the solution for this use case; so long as there is some way to run Rust code during a build which can output files to a path that external tools can predict, that could work.
For the particular cases of CXX(-Qt), this mechanism for generating artifacts should be available to any crate in the dependency graph, not only the final library/executable. It would be a bit inefficient if this was somewhere other than the build script because CXX(-Qt) need to generate and compile the corresponding C++ source file for the output C++ header which needs to be done in the build script, but if there was a bit of redundant code run in both the build script and some artifact-generation-script, that wouldn't be the end of the world.
I would very much rather not frame these discussions more generally; the concrete details for how a solution is wanting to be used and why can make a big difference.
@ssrlive
Here is my code to find out
CARGO_TARGET_DIR
, hope it can help somebody.
Thanks for the solution, but unfortunately it doesn't work with custom profiles
The PROFILE
variable sets to either release
or debug
, regardless of an actual profile name.
Thus, if I use a custom profile, for example:
[profile.production]
inherits = "release"
lto = true
My OUT_DIR
will be /project-dir/target/production/build/crate-name-17eab3514163ed76/out
.
I needed another way to get the target directory, so I thought, why not just skip the required number of directories?
The only caveat is that if I compile a target other than host's, cargo adds a subdirectory with a name of target triple, for example wasm32-unknown-unknown
. So I need to skip one more directory in this case.
I wrote the following code and so far everything works
fn get_cargo_target_dir() -> Result<std::path::PathBuf, Box<dyn std::error::Error>> {
let skip_triple = std::env::var("TARGET")? == std::env::var("HOST")?;
let skip_parent_dirs = if skip_triple { 4 } else { 5 };
let out_dir = std::path::PathBuf::from(std::env::var("OUT_DIR")?);
let mut current = out_dir.as_path();
for _ in 0..skip_parent_dirs {
current = current.parent().ok_or("not found")?;
}
Ok(std::path::PathBuf::from(current))
}
Perhaps there are details that I didn't take or there are ways to make the code more reliable. So let me know if you have any ideas. Regarding the feature discussed, I definitely need a reliable way to get the target directory, so +1
+1 to the use-case of "Staging files [...] next to the cargo-built final artifacts".
In my case I want to copy a config.toml file from my crate root into a new file in the same directory as the the final .exe of the binary.
I want to chime in and support the usecase of "generating header files for FFI" — I'm surprised that this isn't a solved problem. In my case, I could do this at a higher level in the build system (I'm doing FFI between Rust and Dart, and need to get header files for Dart's ffigen program, so I could have a Makefile that calls a binary target in my rust code that generates the headers and puts them in a expected location before calling the dart build or something, but that seems a lot more complicated than just generating the headers when the code is built, which has the nice property of preventing the headers from getting out of sync with the .so
file, for instance)
Wanted to give a reminder from my earlier post
So the way to move this forward is for us to look at the use cases and try to identify solutions without CARGO_TARGET_DIR
For example, I'm assuming we'd be needing some kind of staging directory env variable that we would then copy/move to the --out-dir
(not to be mistaken with OUT_DIR
).
We'd need to deal with how to avoid conflicts with these staged files and what build scripts are allowed to participate (root crate only? workspace members? anything?). The wider the scope, the more likely and the more complicated the solution would be for avoiding conflicts.
For the generating header files for FFI use case, it's important that placing files in this to-be-determined place is not limited to only the root crate because when linking staticlib crates into an executable, only one staticlib is allowed or there will be duplicate symbols from the multiple Rust runtimes (https://hg.mozilla.org/mozilla-central/rev/841c2247f57d).
I think it makes sense to have target_dir available to build scripts. It's not difficult to infer it from OUT_DIR
and so far it's been consistent to be somewhere below target/{debug, release}
.
My use case is to place a vendor DLL next to my binary so that I can load it dynamically at runtime. Typically the default location is next to executable. It's possible to override and load DLL from somewhere else or run custom script to copy files but I think it's neat to have everything in build.rs
rather than provide lengthy build instructions.
Note that directly exposing
CARGO_TARGET_DIR
was attempted in the past but was rejected, see #8710 (comment)
I would like to re-open the premise that "CARGO_TARGET_DIR
cannot be stably provided to client build scripts" for discussion. I know you want us to stop asking for this, but I honestly do not understand why this can't be provided. No matter what happens in the future, cargo build is always going to create binaries and put them in some directory.
Maybe "CARGO_TARGET_DIR" is the wrong name for this directory, but let's ignore the name. Surely there is a directory where cargo is planning to put the binary that it's in the process of building. Why couldn't cargo tell us what it is?
No matter what happens in the future, cargo build is always going to create binaries and put them in some directory.
Here's one counter-example: a future, non-cargo build system consumes crates.io packages, and builds the in a distributed way (packages are built on different machines with intermediate artifacts sent over the network). In this world, there isn't a shared TARGET_DIR.
More generally: the question here is not as much about behavior of cargo
command line tool, but about the specification of what a "crates.io Rust package" is. cargo
is one consumer of such packages, but there already are others (various cargo2X
tools, like cargo raze
), and there will be more in the future.
Adding CARGO_TARGET_DIR
with guarantees expands the interface the packages can rely onto, and:
CARGO_TARGET_DIR
are relied on by the build scripts?For example, https://github.com/rust-lang/cargo/issues/9661#issuecomment-892102871 suggests using CARGO_TARGET_DIR
as a global communication channel between all builds, but that necessary rules out a possibility of distributed builds.
In contrast, the more narrow "Staging files" use-case feels like it can work fine --- each build script can write extra output to its local OUT_DIR, and then inform cargo about this extra output, and ask cargo to copy it over to the final destination, wherever that might be.
Not saying that we should not expose CARGO_TARGET_DIR directly, but I do want to push back against "why don't you just do that already?". In the limit, that strategy is equivalent to make
--- everyone is free to do arbitrary things in their build, but it is no longer possible to automatically combine separate projects, as there's no common interface all dependencies follow.
CARGO_TARGET_DIR as a global communication channel between all builds, but that necessary rules out a possibility of distributed builds.
I also don't think it is a good idea to share CARGO_TARGET_DIR globally. But how about exposing CARGO_TARGET_DIR for each package separately? Build Scripts can only get their own TARGET_DIR, and there is no necessary to share TARGET_DIR in distributed build system.
Would someone be up for summarizing this thread, including the use cases, why OUT_DIR
doesn't work, proposed solutions, and challenges with those proposed solutions?
As it stands, on the surface I don't see us passing CARGO_TARGET_DIR
to build scripts and would be inclined to propose to the Cargo team for us to close it. Having that information could help with splitting out the needed issues or maybe even dissuade us from closing this.
Here's a concrete use-case. I am building a driver which wraps manufacturer-written C code including static libs. These libs are pre-compiled on a per-architecture basis and additionally may not be distributed as part of the crate per the manufacturer's licensing. The desire is to have our wrapper crate which, in its own build.rs file, searches the user workspace's Cargo.toml for something like:
usercrate Cargo.toml
[package.metadata.drivercrate]
staticlibpath = "./proprietarylibs"
Then the drivercrate would be able to use CARGO_TARGET_DIR + $metadata.staticlibpath
to find the user-provided files.
As an aside, embuild does this already, but needs to be added by the user, as a build dependency. Not ideal.
@noahbliss I'm confused why you would want the proprietary libraries inside the target directory in that case. The target directory should be trivial to regenerate by rerunning cargo build
. It's not uncommon for me to delete the entire target directory. So it seems like a strange place to want to put permanent files needed by the crate. I'd suggest using a path relative to the manifest directory (which would be added to .gitignore) or an arbitrary path specified by an environment variable in that case.
Hey @Be-ing sorry I misspoke, I'd be looking for something like CARGO_WORKSPACE_DIR
The issue arising from your suggestion is that the crate which is bringing the headers and functions for these static libs is not the crate the user is working with, it's a dependency. I.e. I have a dependency crate that needs to access files in CARGO_WORKSPACE_DIR + somedirectory
Right now we're doing this through just telling users to set an ENV var in .cargo/config.toml
@noahbliss the issue for that is #3946
Would someone be up for summarizing this thread, including the use cases, why
OUT_DIR
doesn't work, proposed solutions, and challenges with those proposed solutions?As it stands, on the surface I don't see us passing
CARGO_TARGET_DIR
to build scripts and would be inclined to propose to the Cargo team for us to close it. Having that information could help with splitting out the needed issues or maybe even dissuade us from closing this.
The original issue, which is still a challenge for Windows developers, is this:
I have two crates. The one "client" crate depends on the other "server" crate. The server crate is a cdylib
lib crate that produces a DLL and exports a function. The client crate is a simple lib crate with a #[test]
function that needs to load the DLL and call the function.
The client crate has a build dependency on the server crate so that cargo test -p client
will build the server before building the client. This runs into https://github.com/rust-lang/cargo/issues/7825 but at least works.
The problem is that while cargo test -p client
builds the server DLL before building and running the client test, the server DLL is placed in some intermediate directory and the client test has no idea how to find it. Conventionally, the server DLL needs to be placed in the same directory for the Windows loader to find it and mimic xcopy deployment.
Windows toolchains typically provide a common output directory to receive all needed artifacts and a post-build script that can consolidate any additional artifacts manually.
I'm not demanding any given solution, just pointing out that this is a very common workflow on Windows, and I just don't see a way to make it happen reliably with Cargo. Perhaps I've missed something.
@kennykerr so are just needing a way to find the final artifact of a dependency, right?
Would artifact dependencies solve this?
Thank you for clarifying @kennykerr. I think we have at least two distinct use cases here:
The problem is that while cargo test -p client builds the server DLL before building and running the client test, the server DLL is placed in some intermediate directory and the client test has no idea how to find it. Conventionally, the server DLL needs to be placed in the same directory for the Windows loader to find it and mimic xcopy deployment.
To clarify, the artifact the downstream crate needs is the DLL generated by Cargo as part of the normal build process, not custom code in a build script, correct? Are there C/C++ header involved in this process too?
This is a bit different from the C/C++ bindings cases where the artifacts (headers) are generated via custom code in a build script rather than Cargo's normal output.
so are just needing a way to find the final artifact of a dependency, right?
@epage - Yes, although it may be more than one file - for example you typically need both server.dll and server.pdb...
@Be-ing - I am not talking about interop with other build systems or interop with C++. This is all just Rust code and crates. When its all tested and done, I may end up taking the DLL and shipping it to customers outside of the Rust universe but that's a separate issue that is outside the scope of this problem, at least for me.
I will check out artifact dependencies.
I did some experimenting with artifact dependencies. Very promising! I can use the CARGO_CDYLIB_DIR_SERVER
environment variable from the client's build script to find the server DLL but it's still quite a challenge to copy it to the directory where the client executable will be placed. If the client is a test I need to copy two directories up, but if the client is a binary then I need to copy it three directories up. It will probably also be different if I need to test with an alternative --target
. It would be nice to have another build script environment variable for where the client executable will be run from for either cargo run -p client
or cargo test -p client
.
This is required for safe DLL loading on Windows via LoadLibraryExW(w!("server.dll"), 0, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS)
.
Should anyone else stumble upon this, here's an example.
[package]
name = "server"
edition = "2021"
[lib]
crate-type = ["cdylib"]
#[no_mangle]
unsafe extern "system" fn server() -> i32 {
println!("hello server");
123
}
[package]
name = "client"
edition = "2021"
[build-dependencies.server]
path = "../server"
artifact = "cdylib"
[dependencies.windows-sys]
version = "0.52"
features = ["Win32_Foundation", "Win32_System_LibraryLoader"]
fn main() {
let var = std::env::var("CARGO_CDYLIB_DIR_SERVER").expect("var not found");
let from = std::path::Path::new(&var);
let to = from
.parent()
.expect("parent not found")
.parent()
.expect("parent not found")
.parent()
.expect("parent not found");
for entry in std::fs::read_dir(from).expect("from not found") {
let from = entry.expect("entry not found").path();
std::fs::copy(
&from,
to.with_file_name(from.file_name().expect("file name not found")),
)
.expect("copy failed");
}
}
fn main() {
unsafe {
use windows_sys::{core::*, Win32::System::LibraryLoader::*};
let library = LoadLibraryExW(w!("server.dll"), 0, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
let address = GetProcAddress(library, s!("server")).expect("server not found");
let server: extern "system" fn() -> i32 = std::mem::transmute(address);
assert_eq!(server(), 123);
}
}
Run as follows: cargo run -p client -Z bindeps
It's the client build script that continues to be problematic.
Is there a way to reliably determine where Cargo will place the client binary/test executable?
What we do in debug builds is set the env variable and use it to locate the DLL, in release builds we rely on executable dir.
@kennykerr maybe the dynamic library paths that can be set from your build.rs build script via cargo::rustc-link-search
could work without having to copy the DLL for cargo run
and cargo test
?
The problem: I need to locate the target dir from the build script.
The solution: I'd love a
CARGO_TARGET_DIR
environment variable. There's clearly precedent for this ascargo doc
depends on such a thing today.@jyn514 suggested I reach out to the Cargo team for advice.
Currently the Windows crate has been using various hacky solutions that are problematic.
https://github.com/rust-lang/docs.rs/issues/1444#issuecomment-875188411