CMake fails on Windows when any involved path is over 260 characters in length

[matthew-e-brown]

I'm trying to use the GLFW crate to work with OpenGL. By default, that crate runs CMake as part of a build script to compile GLFW, which is the desired behaviour (as opposed to linking to a pre-compiled version). The GLFW crate relies on a secondary crate called glfw-sys, which as far as I can tell is just the regular GLFW but with a super simple build.rs script that, in turn, uses this crate to set some configuration and then run CMake.

The only thing I'm unsure of is whether or not this bug report should be made here, or if it should be made all the way downstream at CMake's official repository. Please feel free to shove me over to them if you feel this issue has nothing to do with this Rust wrapper. 😁

Backstory

After installing CMake (version 3.25.0) and putting it on my PATH, I followed the guide on GLFW's site combined with LearnOpenGL to get a window up and running, and it works totally fine:

(The code for this example is available as a GitHub Gist, if you wish to see it)

"Excellent," I thought to myself, "now I'm ready to start the final project for my Computer Graphics course" (we've been using WebGL all semester, I figured I should try real OpenGL). However, when I duplicate the project into my folder for my course, things break:

The full output is as follows:

   Compiling cfg-if v1.0.0
   Compiling khronos_api v3.1.0
   Compiling winapi v0.3.9
   Compiling log v0.4.17
   Compiling cc v1.0.76
   Compiling xml-rs v0.8.4
   Compiling windows_x86_64_msvc v0.42.0
   Compiling time-core v0.1.0
   Compiling time-macros v0.2.6
   Compiling itoa v1.0.4
   Compiling cty v0.2.2
   Compiling cmake v0.1.49
   Compiling gl_generator v0.14.0
   Compiling lazy_static v1.4.0
   Compiling raw-window-handle v0.4.3
   Compiling windows-sys v0.42.0
   Compiling bitflags v1.3.2
   Compiling time v0.3.17
   Compiling gl v0.14.0
   Compiling glfw-sys v4.0.0+3.3.5
   Compiling atty v0.2.14
   Compiling colored v2.0.0
   Compiling simple_logger v4.0.0
error: failed to run custom build command for `glfw-sys v4.0.0+3.3.5`

Caused by:
  process didn't exit successfully: `C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-e2e03b0e00e0d338\build-script-build` (exit code: 101)
  --- stdout
  CMAKE_TOOLCHAIN_FILE_x86_64-pc-windows-msvc = None
  CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc = None
  HOST_CMAKE_TOOLCHAIN_FILE = None
  CMAKE_TOOLCHAIN_FILE = None
  CMAKE_GENERATOR_x86_64-pc-windows-msvc = None
  CMAKE_GENERATOR_x86_64_pc_windows_msvc = None
  HOST_CMAKE_GENERATOR = None
  CMAKE_GENERATOR = None
  CMAKE_PREFIX_PATH_x86_64-pc-windows-msvc = None
  CMAKE_PREFIX_PATH_x86_64_pc_windows_msvc = None
  HOST_CMAKE_PREFIX_PATH = None
  CMAKE_PREFIX_PATH = None
  CMAKE_x86_64-pc-windows-msvc = None
  CMAKE_x86_64_pc_windows_msvc = None
  HOST_CMAKE = None
  CMAKE = None
  running: "cmake" "C:\\Users\\Matthew\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\glfw-sys-4.0.0+3.3.5\\." "-G" "Visual Studio 17 2022" "-Thost=x64" "-Ax64" "-DGLFW_BUILD_EXAMPLES=OFF" "-DGLFW_BUILD_TESTS=OFF" "-DGLFW_BUILD_DOCS=OFF" "-DCMAKE_INSTALL_LIBDIR=lib" "-DGLFW_BUILD_WAYLAND=ON" "-DCMAKE_INSTALL_PREFIX=C:\\Users\\Matthew\\Dropbox\\Coursework\\5th Year - 2022-2023\\Fall\\COIS-4480H Computer Graphics\\FinalProject\\target\\debug\\build\\glfw-sys-9e8b97620887df2c\\out" "-DCMAKE_C_FLAGS= -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Debug"
  -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19045.
  -- The C compiler identification is MSVC 19.33.31630.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - failed
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe - broken
  -- Configuring incomplete, errors occurred!
  See also "C:/Users/Matthew/Dropbox/Coursework/5th Year - 2022-2023/Fall/COIS-4480H Computer Graphics/FinalProject/target/debug/build/glfw-sys-9e8b97620887df2c/out/build/CMakeFiles/CMakeOutput.log".
  See also "C:/Users/Matthew/Dropbox/Coursework/5th Year - 2022-2023/Fall/COIS-4480H Computer Graphics/FinalProject/target/debug/build/glfw-sys-9e8b97620887df2c/out/build/CMakeFiles/CMakeError.log".

  --- stderr
  CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
    The C compiler

      "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe"

    is not able to compile a simple test program.

    It fails with the following output:

      Change Dir: C:/Users/Matthew/Dropbox/Coursework/5th Year - 2022-2023/Fall/COIS-4480H Computer Graphics/FinalProject/target/debug/build/glfw-sys-9e8b97620887df2c/out/build/CMakeFiles/CMakeScratch/TryCompile-4wccup

      Run Build Command(s):C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/amd64/MSBuild.exe cmTC_187a6.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=17.0 /v:m && MSBuild version 17.3.1+2badb37d1 for .NET Framework
        testCCompiler.c
      FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.dir\Debug\cmTC_187a6.tlog\link-cvtres.read.1.tlog. The system cannot find the path specified. [C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.vcxproj]

      FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.dir\Debug\cmTC_187a6.tlog\link-cvtres.write.1.tlog. The system cannot find the path specified. [C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.vcxproj]

    CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
    CMakeLists.txt:3 (project)

  thread 'main' panicked at '
  command did not execute successfully, got: exit code: 1

  build script failed, must exit now', C:\Users\Matthew\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.49\src\lib.rs:1104:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Explanation

From reading the error message, it seem that the crux of the issue is a part of CMake Microsoft's cl.exe called FileTracker. The key parts seem to be the following lines:

FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.dir\Debug\cmTC_187a6.tlog\link-cvtres.read.1.tlog. The system cannot find the path specified. [C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.vcxproj]

FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.dir\Debug\cmTC_187a6.tlog\link-cvtres.write.1.tlog. The system cannot find the path specified. [C:\Users\Matthew\Dropbox\Coursework\5th Year - 2022-2023\Fall\COIS-4480H Computer Graphics\FinalProject\target\debug\build\glfw-sys-9e8b97620887df2c\out\build\CMakeFiles\CMakeScratch\TryCompile-4wccup\cmTC_187a6.vcxproj]

I am fairly certain that the problem is the path length, since the only difference between the two attempts is the location. I've tried several: ~/Desktop/Some Folder/opengl works fine, so it's not the spaces; ~/Dropbox/Code/Tests/rust-opengl works fine, so it's not the fact that it's inside Dropbox. Most damingly, though, is the fact that simply renaming FinalProject to Project fixes things.

I have long been aware of Windows' 260-character limit for file paths. Upon closer inspection, even though my project directory isn't that deeply nested, all of the build folders (glfw-sys/out/build/CMakeFiles/.../Debug/cmTC_187a6.tlog/...tlog) push the final total length for those FileTracker paths to an incredibly unlucky 261 characters. I did try to disable the limit, but that sadly didn't help (I think disabling it only works if your applications use the funky \\?\ format).

Here's a full screenshot of the target/debug/build's glfw-sys-* directories. If you would like to see any other log files let me know and I'll send them. Here are the two files it says to "see also", CMakeError.log and CMakeOutput.log.

Conclusion

As I said at the start, I'm not sure if this issue is from this crate, or if it's a problem with CMake directly. If you ask me, it's probably an issue with CMake itself, since I'd assume that the program actually interacting with the FS should be the one prefixing the \\?\ things to paths longer than 260. However, I'm also pretty sure that (since CMake is probably just using the paths as passed to it directly) if this crate were to do the prefixing, it would work.

I figured it couldn't hurt to open an issue here first and then go to CMake's main repo if this crate's maintainers found it to be out-of-scope.

Thanks.

[ChrisDenton]

Is FileTracker part of CMake itself or a separate thing? In any case, yeah this is ultimately an issue with whatever thing is accessing the filesystem. Rust itself should handle long paths automatically in fs operations, therefore cmake-rs should too by default.

In general, Rust won't automatically make \\?\ paths when passing paths to other applications because many applications don't handle them very well (even some that can use long paths internally). I guess we could try canonicalizing all paths (which adds the \\?\) but I'd be worried about unintended side-effects in complex build systems.

[thomcc]

It's a bit odd but I think we'd be willing to do that if the cmake maintainers say that that's the approach we should take.

But short of that (or some other cmake expert saying its the right approach), I'm inclined to say it sounds like a pretty idiosyncratic thing to do that would be likely to cause problems rather than fix them...

[ChrisDenton]

In short, I'd suggest reporting this to CMake but it's quite possible they'll pass you on to some other project or punt it back to us with some advice on what we should be doing differently.

Unfortunately I don't know CMake's internals well enough to offer any other suggestions but I did have the impression it had at least some support for long paths internally.

[matthew-e-brown]

After doing a bit of looking in the morning with a fresh set of eyes, it looks like FileTracker is not a part of CMake at all, but instead a part of the C compiler that CMake finds: it's a Microsoft thing from the "Visual Studio 17 2022" cl.exe that it's using.

Don't know how I didn't catch that last night. CMake says pretty clearly,

The C compiler .../MSVC/14.33.31629/bin/Hostx64/x64/cl.exe is not able to compile a simple test program. It fails with the output: ... (FileTracker Errors)

So... now I'm unsure as to where to bring this. I don't think I should really be all too surprised that Microsoft's own tooling doesn't support their own long paths, but I can't find "FileTracker" online anywhere except for that one link I used above.

I think what I'll do for now is bring this up in the CMake repository and see if any of them have any better ideas as far as what branch of what chain to take this up. Some class in some project doesn't support long paths, but it's gonna be a nightmare to get this up that chain 😆

[strega-nil]

@matthew-e-brown yeah, this is one of those unfortunate things about working on Windows; long paths are not well supported by legacy programs, and there aren't many programs more legacy than the Microsoft Visual C++ Compiler (my understanding is that the compiler MSVC is based on was first released in June of 1982).

I would recommend putting your project at a shallower directory structure; there's just not much else to do. If it helps, what I do is create a virtual disk which I then mount as a new drive; this is kind of the only way to get Windows builds to work, for example.

[matthew-e-brown]

Thanks @strega-nil. That's exactly what I did. At first it was incredibly unlucky that the path the FileTracker tries to write to was 261 characters, exactly one longer than supported—so close to avoiding the error. But now that it's happened, I'm incredibly lucky that it was only one character over: I simply changed 'FinalProject' to 'Project' and everything's working fine now, just in time for me to meet my deadline for my Graphics course 😄

I'll look into filing an "issue" (or whatever the equivalent is) to Microsoft for the C++ build tools, but I don't think I'll persue it too passionately. Given its age I doubt it would be easy to patch (I knew it was old, but I didn't know it was that old 😳).

I'll close this issue here for now, but come back and re-open it if I ever get anywhere with Microsoft (if I do end up sending them anything at all). Cheers.

[strega-nil]

Yeah, it's not even close to one of our priorities. Long paths are neat, but they're not worth making massive changes to the compiler for (which is likely what we'd have to do). If you do still want to file an issue, Developer Community is the place to go.

Glad you have a workaround that works for you!