search

bazelbuild / rules_rust

Rust rules for Bazel
https://bazelbuild.github.io/rules_rust/
Apache License 2.0
652 stars 412 forks source link

Systematically handle Windows MAX_PATH issues #1120

Open UebelAndre opened 2 years ago

UebelAndre commented 2 years ago

I feel like I run into Windows MAX_PATH issues more often in rules_rust than I do for other rules. I'm wondering if there's a way to address this in a systematic way using the process_wrapper or something. But it'd be nice to not need to worry about this since there's apparently solutions (\\?\ prefixes) that might be able to help.

hlopko commented 2 years ago

So if I understand right - the issue is that rules_rust produce paths that are too long on Windows. We have multiple solutions:

1) decreasing the chance of this being a problem by using shorter names (for example by using crate_universe_examples instead of rules_rust_examples_crate_universe) 2) implementing support for \\?\ in the process wrapper 3) implementing support for \\?\ in starlark implementation of rules

Let's use this issue to collect more data first. Which paths are problematic? Tools? Files?

UebelAndre commented 2 years ago

https://github.com/bazelbuild/rules_rust/pull/1006 was another example of this issue

UebelAndre commented 2 years ago

https://github.com/bazelbuild/rules_rust/issues/423 is also an example of this issue.

UebelAndre commented 2 years ago

https://github.com/bazelbuild/rules_rust/pull/1420#issuecomment-1180371623 shows another example of this issue.

scentini commented 2 years ago

https://github.com/bazelbuild/rules_rust/pull/1434 shortened a target name for the same reason too.

So far, whenever I've ran into this issue, the culprit turned out to be MSVC, especially the linker, and I don't believe we can do anything reasonable about that (maybe in process_wrapper we could copy all the files to a temporary location and copy the output file to the desired location afterwards, but that would be expensive). Bazel's best practices suggest using a short output_user_root but we already do that on CI and are still close to the limit even for some toolchain files.

I believe one of the reasons we are hitting this issue sooner than other rulesets are the temporary files that rustc produces, eg

bazel build --spawn_strategy=local  --//:extra_rustc_flags=-Csave-temps test/unit/native_deps:proc_macro_has_native_dep

will create a proc_macro_has_native_dep-1976543513.proc_macro_using_native_dep.26a1bb4a-cgu.1.rcgu.o file. I don't think there is a way to instruct rustc to shorten this filename.

Last time I talked to Bazel's Windows expert @meteorcloudy I got the impression that there is not much we can do about MSVC, but tagging them to confirm/contradict me.

meteorcloudy commented 2 years ago

@scentini You're right, we cannot do anything about it other than try to use a shorter path. The MSVC compiler/link doesn't work with the \\?\ solution.

This issue is reported back to Microsoft a long time ago: https://developercommunity.visualstudio.com/t/allow-building-running-and-debugging-a-net-applica/351628

UebelAndre commented 1 year ago

https://github.com/bazelbuild/rules_rust/issues/1556 seems to be another case of this issue.

tomgr commented 10 months ago

I've run into this issue several times when updating rules_rust recently on our project. I've worked around this by using a patch to rules_rust that truncates the auto-generated workspace names so that a workspace name like:

@rust_windows_x86_64__x86_64-pc-windows-msvc__stable

becomes

@rwx6x6pwms

(It's just taking the first character of each _ separated component.) Although this is horribly ugly, it has reliably solved the problem for us and we've not had any more path issues on any of the crates we're compiling. The patch is not invasive - there are only two places where this truncation is required.

Given that this problem appears unlikely to be solvable without MSVC supporting long paths, would a PR for such a truncation scheme (enabled only on Windows) be useful? I appreciate it's not pretty, but I don't see many other options without MSVC supporting long paths. The only other option would seem to be copying files into a short temporary location, but I expect this would be expensive.

UebelAndre commented 10 months ago

Given that this problem appears unlikely to be solvable without MSVC supporting long paths, would a PR for such a truncation scheme (enabled only on Windows) be useful? I appreciate it's not pretty, but I don't see many other options without MSVC supporting long paths. The only other option would seem to be copying files into a short temporary location, but I expect this would be expensive.

Shortening repository names to short ugly paths is what we've been doing to solve this in other places so yeah, if there is a way to enable that for windows then that sounds good to me! Just recently I was wondering if such a thing were possible. Perhaps the verbose repository name could still be created but it only contains aliases to the ugly one so for building short paths get used by folks can still refer to the verbose one? Not sure if that's that valuable though. Working builds definitely are though 😄

peakschris commented 2 months ago

@tomgr (or anyone else who has it) - could you share your horrible hack? I'm running into this issue. Thank you!