rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.02k stars 12.68k forks source link

Spurious failures on AppVeyor due to ar.exe errors #40546

Closed alexcrichton closed 7 years ago

alexcrichton commented 7 years ago

I've previously been classifying these errors under https://github.com/rust-lang/rust/issues/40240 but now I'm thinking that's incorrect, I suspect these are totally separate errors.

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2397

[ 13%] Linking CXX static library ../../libLLVMMCDisassembler.a
C:/projects/rust/mingw64/bin/ar.exe: ../../libLLVMMCDisassembler.a: Permission denied
make[2]: *** [lib/MC/MCDisassembler/CMakeFiles/LLVMMCDisassembler.dir/build.make:205: lib/libLLVMMCDisassembler.a] Error 1
make[2]: *** Deleting file 'lib/libLLVMMCDisassembler.a'
make[1]: *** [CMakeFiles/Makefile2:2539: lib/MC/MCDisassembler/CMakeFiles/LLVMMCDisassembler.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 13%] Built target LLVMMCParser
make: *** [Makefile:150: all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2
alexcrichton commented 7 years ago

The first hit on Google is quite ominous. My preference for an attempt to fix this would be to switch to ninja as it's likely just flat out more reliable than makefiles on MinGW

alexcrichton commented 7 years ago

Oh also we're going to want to switch the MSVC build to Ninja regardless, so we'll have it available anyway eventually on the bots.

mati865 commented 7 years ago

I've met it few times with MSYS2 and (almost?) always it was caused by CL RF line endings or too long paths or file names. When compiling Rust from git I found there are files that windows explorer cannot remove since they are too long. Unfortunately I don't have list as I used MSYS2 rm -rf * to clean those files.

alexcrichton commented 7 years ago

Unfortunately it looks like this is not fixed

alexcrichton commented 7 years ago

Another attempt to fix this

alexcrichton commented 7 years ago

Nope, still not fixed

mati865 commented 7 years ago

You can try changing temp directory if it is possible with rustbuild.

Dnia 29 marca 2017 19:01:24 CEST, Alex Crichton notifications@github.com napisał(a):

Nope, still not fixed

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/rust-lang/rust/issues/40546#issuecomment-290154911

-- Wysłane za pomocą K-9 Mail.

aidanhs commented 7 years ago

Just looked it up because I was curious - in theory appveyor uses core.autocrlf input (i.e. newlines are always checkout out as LF), which is the suggested solution in the SO post. So no help there.

carols10cents commented 7 years ago

Does not look fixed to me :( My untrained eyes aren't seeing much of a pattern though...

3 say "No such file", 3 say "Permission denied".

logs form building #40971:

[1353/1629] Linking CXX static library lib\libLLVMJSBackendInfo.a
FAILED: lib/libLLVMJSBackendInfo.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMJSBackendInfo.a && C:\projects\rust\mingw64\bin\ar.exe qc lib\libLLVMJSBackendInfo.a  lib/Target/JSBackend/TargetInfo/CMakeFiles/LLVMJSBackendInfo.dir/JSBackendTargetInfo.cpp.obj && C:\projects\rust\mingw64\bin\ranlib.exe lib\libLLVMJSBackendInfo.a && cd ."
C:\projects\rust\mingw64\bin\ar.exe: unable to rename 'lib\libLLVMJSBackendInfo.a'; reason: No such file or directory
[1354/1629] Linking CXX static library lib\libLLVMCoverage.a
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 332.638
Build completed unsuccessfully in 0:07:56
Command exited with code 101

logs from building #40702:

[1070/1629] Linking CXX static library lib\libLLVMAArch64Disassembler.a
FAILED: lib/libLLVMAArch64Disassembler.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMAArch64Disassembler.a && C:\projects\rust\mingw64\bin\ar.exe qc lib\libLLVMAArch64Disassembler.a  lib/Target/AArch64/Disassembler/CMakeFiles/LLVMAArch64Disassembler.dir/AArch64Disassembler.cpp.obj lib/Target/AArch64/Disassembler/CMakeFiles/LLVMAArch64Disassembler.dir/AArch64ExternalSymbolizer.cpp.obj && C:\projects\rust\mingw64\bin\ranlib.exe lib\libLLVMAArch64Disassembler.a && cd ."
C:\projects\rust\mingw64\bin\ar.exe: unable to rename 'lib\libLLVMAArch64Disassembler.a'; reason: No such file or directory
[1071/1629] Linking CXX static library lib\libLLVMAArch64AsmPrinter.a
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 264.200
Build completed unsuccessfully in 0:06:26
Command exited with code 101

logs from building #41121:

[1317/1629] Linking CXX static library lib\libLLVMSystemZDisassembler.a
FAILED: lib/libLLVMSystemZDisassembler.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMSystemZDisassembler.a && C:\projects\rust\mingw64\bin\ar.exe qc lib\libLLVMSystemZDisassembler.a  lib/Target/SystemZ/Disassembler/CMakeFiles/LLVMSystemZDisassembler.dir/SystemZDisassembler.cpp.obj && C:\projects\rust\mingw64\bin\ranlib.exe lib\libLLVMSystemZDisassembler.a && cd ."
C:\projects\rust\mingw64\bin\ar.exe: unable to rename 'lib\libLLVMSystemZDisassembler.a'; reason: No such file or directory
[1318/1629] Linking CXX static library lib\libLLVMSystemZAsmParser.a
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 357.723
Build completed unsuccessfully in 0:08:28
Command exited with code 101

logs from building #40873:

[1432/1629] Building CXX object lib/Target/MSP430/InstPrinter/CMakeFiles/LLVMMSP430AsmPrinter.dir/MSP430InstPrinter.cpp.obj
DEBUG:sccache::commands: Server sent CompileStarted
[1433/1629] Linking CXX static library lib\libLLVMMSP430AsmPrinter.a
FAILED: lib/libLLVMMSP430AsmPrinter.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMMSP430AsmPrinter.a && C:\projects\rust\mingw32\bin\ar.exe qc lib\libLLVMMSP430AsmPrinter.a  lib/Target/MSP430/InstPrinter/CMakeFiles/LLVMMSP430AsmPrinter.dir/MSP430InstPrinter.cpp.obj && C:\projects\rust\mingw32\bin\ranlib.exe lib\libLLVMMSP430AsmPrinter.a && cd ."
C:\projects\rust\mingw32\bin\ar.exe: unable to rename 'lib\libLLVMMSP430AsmPrinter.a'; reason: Permission denied
[1434/1629] Linking CXX static library lib\libLLVMMSP430CodeGen.a
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 380.247
Build completed unsuccessfully in 0:08:32
Command exited with code 101

logs from building #41121:

[1119/1629] Linking CXX static library lib\libLLVMAArch64Utils.a
FAILED: lib/libLLVMAArch64Utils.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMAArch64Utils.a && C:\projects\rust\mingw64\bin\ar.exe qc lib\libLLVMAArch64Utils.a  lib/Target/AArch64/Utils/CMakeFiles/LLVMAArch64Utils.dir/AArch64BaseInfo.cpp.obj && C:\projects\rust\mingw64\bin\ranlib.exe lib\libLLVMAArch64Utils.a && cd ."
C:\projects\rust\mingw64\bin\ranlib.exe: unable to rename 'lib\libLLVMAArch64Utils.a'; reason: Permission denied
[1120/1629] Building CXX object lib/Target/ARM/CMakeFiles/LLVMARMCodeGen.dir/ARMAsmPrinter.cpp.obj
DEBUG:sccache::commands: Server sent CompileStarted
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 302.411
Build completed unsuccessfully in 0:07:34
Command exited with code 101

logs from building #40878:

FAILED: lib/libLLVMMipsDesc.a 
cmd.exe /C "cd . && "C:\Program Files (x86)\CMake\bin\cmake.exe" -E remove lib\libLLVMMipsDesc.a && C:\projects\rust\mingw64\bin\ar.exe qc lib\libLLVMMipsDesc.a  lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsABIInfo.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsABIFlagsSection.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsAsmBackend.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsELFObjectWriter.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsELFStreamer.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsMCAsmInfo.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsMCCodeEmitter.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsMCExpr.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsMCTargetDesc.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsNaClELFStreamer.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsOptionRecord.cpp.obj lib/Target/Mips/MCTargetDesc/CMakeFiles/LLVMMipsDesc.dir/MipsTargetStreamer.cpp.obj && C:\projects\rust\mingw64\bin\ranlib.exe lib\libLLVMMipsDesc.a && cd ."
C:\projects\rust\mingw64\bin\ranlib.exe: lib\libLLVMMipsDesc.a: Permission denied
[1274/1629] Linking CXX static library lib\libLLVMMipsAsmParser.a
ninja: build stopped: subcommand failed.
thread 'main' panicked at '
command did not execute successfully, got: exit code: 1
build script failed, must exit now', C:\Users\appveyor\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.22\src\lib.rs:617
note: Run with `RUST_BACKTRACE=1` for a backtrace.
    finished in 308.299
Build completed unsuccessfully in 0:07:12
Command exited with code 101
alexcrichton commented 7 years ago

I ran the LLVM build a few dozen times locally and it failed lots of times with 6.2.0 (that we're using). I re-verified and it has yet to fail with 6.3.0.

Some next steps:

alexcrichton commented 7 years ago

Does anyone know where the source code and/or build configuration are for the toolchains we're downloading? Best I can find there is 6.1.0 and nothing afterwards...

mati865 commented 7 years ago

@alexcrichton that's a good question. Guess you will have to ask niXman who releases prebuilt packages: https://sourceforge.net/p/mingw-w64/mailman/message/35759499/

arielb1 commented 7 years ago

@alexcrichton

According to this cygwin thread, the problem is the new gdb. Could you try using the gdb from 6.2.0 and everything else from 6.3.0?

mati865 commented 7 years ago

@arielb1 when you install everything from mingw-builds everything is bundled. MSYS2 in turn provides packages but they require recent libraries to work properly, however it is possible to build own package against old libraries but it would have to be self hosted by Rust team.

arielb1 commented 7 years ago

@mati865

We could bundle the "known-good" old gdb and copy it or something.

crlf0710 commented 7 years ago

Why don't the rust team just use or patch the MINGW(not MSYS) llvm provided by MSYS2 project? It's absolutely much easier to use and upgrade, right? (They've even got llvm 4.0 ready now...)

They use PKGBUILDs to build the packages.

https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-clang/

It seems there's quite a few patches needed though.

(They've also got a PKGBUILD for mingw-w64-rust... and shipping prebuilt rust 1.16)

alexcrichton commented 7 years ago

@arielb1 yes I think that's the best solution here.

@mati865 ah we don't want to rely on the msys packages as well b/c we can't pin versions. They'll change from under our feet which runs the risk of breaking the build which we'd prefer to avoid.

@crlf0710 this isn't related to LLVM, it's related to the gcc/gdb toolchain versions

Mark-Simulacrum commented 7 years ago

This is, unfortunately, still a problem. See https://github.com/rust-lang/rust/pull/41580#issuecomment-298167916.

Mark-Simulacrum commented 7 years ago

Possibly a false alarm. #41580 is targeting the beta branch, which presumably does not have the fix for this failure, in which case it's "normal" for it to fail spuriously. Will leave open for someone to confirm, though.

alexcrichton commented 7 years ago

@Mark-Simulacrum ah yeah I think we're ok, https://github.com/rust-lang/rust/pull/41420, the fix, is only on master not on beta