search

Argyle-Software / kyber

A rust implementation of the Kyber post-quantum KEM
https://docs.rs/pqc_kyber/
Apache License 2.0
164 stars 37 forks source link

Fails to compile on Windows (x86_64-pc-windows-msvc) #18

Closed faern closed 1 year ago

faern commented 1 year ago

Probably very related to #2. I just clone the master branch and run cargo build and I get the following:

log ``` $ cargo build Compiling pqc_kyber v0.2.1 (C:\Users\faern\src\kyber-master) Compiling synstructure v0.12.6 The following warnings were emitted during compilation: warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/basemul.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/basemul.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/fq.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/fq.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/invntt.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/invntt.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/ntt.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/ntt.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/shuffle.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/shuffle.S' ignored error: failed to run custom build command for `pqc_kyber v0.2.1 (C:\Users\faern\src\kyber-master)` Caused by: process didn't exit successfully: `C:\Users\faern\src\kyber-master\target\debug\build\pqc_kyber-de0e4d6bdeecf67b\build-script-build` (exit code: 1) --- stdout TARGET = Some("x86_64-pc-windows-msvc") OPT_LEVEL = Some("0") HOST = Some("x86_64-pc-windows-msvc") CC_x86_64-pc-windows-msvc = None CC_x86_64_pc_windows_msvc = None HOST_CC = None CC = None CFLAGS_x86_64-pc-windows-msvc = None CFLAGS_x86_64_pc_windows_msvc = None HOST_CFLAGS = None CFLAGS = None CRATE_CC_NO_DEFAULTS = None CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2") DEBUG = Some("true") running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "-c" "src/avx2/basemul.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/basemul.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/basemul.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "-c" "src/avx2/fq.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/fq.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/fq.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "-c" "src/avx2/invntt.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/invntt.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/invntt.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "-c" "src/avx2/ntt.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/ntt.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/ntt.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" "-c" "src/avx2/shuffle.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/shuffle.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/shuffle.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 AR_x86_64-pc-windows-msvc = None AR_x86_64_pc_windows_msvc = None HOST_AR = None AR = None running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\lib.exe" "-out:C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\libpqc_kyber.a" "-nologo" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" LINK : fatal error LNK1181: cannot open input file 'C:\Users\faern\src\kyber-master\target\debug\build\pqc_kyber-4e4cd6b4fd1f06c6\out\src\avx2\basemul.o' exit code: 1181 --- stderr error occurred: Command "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\lib.exe" "-out:C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\libpqc_kyber.a" "-nologo" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" with args "lib.exe" did not execute successfully (status code exit code: 1181). warning: build failed, waiting for other jobs to finish... ```

It works if I build for the gnu target (x86_64-pc-windows-gnu), but that's not the target I build my program for. It of course also works if activating the reference feature.

mberry commented 1 year ago

Can you confirm this is built with:

RUSTFLAGS="-C target-cpu=native -C target-feature=+aes,+avx2,+sse2,+sse4.1,+bmi2,+popcnt"

before compiling?

In your output there is: CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2") and that's not enough for the optimised version to compile.

At the moment anything x86_64 will use the optimised version. This is changed now to: target_feature="avx2"on the macos-fix branch. Master assumes everyone with x86_64 can run avx2 simd instructions. This makes it unnecessary to export the rustflags but includes all the chips that intel somehow still ships without it like modern pentiums and celerons which need to use reference. Perhaps needs some note under #[cfg(debug_assertions)] in the build script to publish whether they are building the reference or optimised version.

faern commented 1 year ago

I get the same error with RUSTFLAGS="-C target-cpu=native -C target-feature=+aes,+avx2,+sse2,+sse4.1,+bmi2,+popcnt".

In your output there is: CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2") and that's not enough for the optimised version to compile.

Then I don't think the crate should enable the optimized version. IMO it would be most developer friendly if the crate always works and optimizations are opt-in instead of opt-out. If they can be automatically enabled in a reliable way that would of course be the best :ok_hand:

New log ``` $ RUSTFLAGS="-C target-cpu=native -C target-feature=+aes,+avx2,+sse2,+sse4.1,+bmi2,+popcnt" cargo build Compiling proc-macro2 v1.0.43 Compiling quote v1.0.21 Compiling unicode-ident v1.0.4 Compiling syn v1.0.100 Compiling unicode-xid v0.2.4 Compiling cc v1.0.73 Compiling cfg-if v1.0.0 Compiling getrandom v0.2.7 Compiling rand_core v0.6.4 Compiling pqc_kyber v0.2.1 (C:\Users\faern\src\kyber-master) The following warnings were emitted during compilation: warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/basemul.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/basemul.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/fq.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/fq.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/invntt.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/invntt.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/ntt.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/ntt.S' ignored warning: cl : Command line warning D9024 : unrecognized source file type 'src/avx2/shuffle.S', object file assumed warning: cl : Command line warning D9027 : source file 'src/avx2/shuffle.S' ignored error: failed to run custom build command for `pqc_kyber v0.2.1 (C:\Users\faern\src\kyber-master)` Caused by: process didn't exit successfully: `C:\Users\faern\src\kyber-master\target\debug\build\pqc_kyber-de0e4d6bdeecf67b\build-script-build` (exit code: 1) --- stdout TARGET = Some("x86_64-pc-windows-msvc") OPT_LEVEL = Some("0") HOST = Some("x86_64-pc-windows-msvc") CC_x86_64-pc-windows-msvc = None CC_x86_64_pc_windows_msvc = None HOST_CC = None CC = None CFLAGS_x86_64-pc-windows-msvc = None CFLAGS_x86_64_pc_windows_msvc = None HOST_CFLAGS = None CFLAGS = None CRATE_CC_NO_DEFAULTS = None CARGO_CFG_TARGET_FEATURE = Some("adx,aes,avx,avx2,bmi1,bmi2,fma,fxsr,lzcnt,pclmulqdq,popcnt,rdrand,rdseed,sse,sse2,sse3,sse4.1,sse4.2,ssse3,xsave,xsavec,xsaveopt,xsaves") DEBUG = Some("true") running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "-c" "src/avx2/basemul.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/basemul.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/basemul.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "-c" "src/avx2/fq.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/fq.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/fq.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "-c" "src/avx2/invntt.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/invntt.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/invntt.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "-c" "src/avx2/ntt.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/ntt.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/ntt.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "src/avx2/" "-W4" "-FoC:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" "-c" "src/avx2/shuffle.S" cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/avx2/shuffle.S', object file assumed cargo:warning=cl : Command line warning D9027 : source file 'src/avx2/shuffle.S' ignored cl : Command line warning D9021 : no action performed exit code: 0 AR_x86_64-pc-windows-msvc = None AR_x86_64_pc_windows_msvc = None HOST_AR = None AR = None running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\lib.exe" "-out:C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\libpqc_kyber.a" "-nologo" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" LINK : fatal error LNK1181: cannot open input file 'C:\Users\faern\src\kyber-master\target\debug\build\pqc_kyber-4e4cd6b4fd1f06c6\out\src\avx2\basemul.o' exit code: 1181 --- stderr error occurred: Command "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\lib.exe" "-out:C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\libpqc_kyber.a" "-nologo" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/basemul.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/fq.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/invntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/ntt.o" "C:\\Users\\faern\\src\\kyber-master\\target\\debug\\build\\pqc_kyber-4e4cd6b4fd1f06c6\\out\\src/avx2/shuffle.o" with args "lib.exe" did not execute successfully (status code exit code: 1181). warning: build failed, waiting for other jobs to finish... ```

EDIT: I also upgraded to VS2022 between these builds. But I don't think that's relevant to the issue at hand(?)

mberry commented 1 year ago

Thanks, those asm symbols should be fine with 64-bit windows but we don't have a test harness for it yet. I'll try to find the cause later this week.

optimizations are opt-in instead of opt-out

Fair call. There's a usability vs "doing it the rust way" tradeoff that I'm mulling over a lot, for an extreme example of how messy it gets when going full generic:

https://github.com/Argyle-Software/sphincsplus/blob/refactor/src/sign.rs#L17-L33

The alternative is large amounts of code duplication and build scripts for dozens of individual crates. As far as trade-off's go it's not an easy choice but this stuff should probably be a different issue altogether.

mberry commented 1 year ago

Got this to build on x86_64-pc-windows-gnu using mingw64 gcc[1]. Alas it's segfaulting at runtime, the assembly is written in GAS and windows will need MASM or NASM. There's not a particularly large amount of it, so it could be rewritten eventually.

I wrongly assumed the optimised code used in the C reference codebase compiled for windows but it doesn't. There's still a lot of optimised code written in rust so I'm thinking the best way forward at this point is to put in target_vendor checks and use the reference ntt functions, it will need a bit of refactoring.

mberry commented 1 year ago

With the version bump avx2 will be opt-in only, so I'll consider this fixed, I'd prefer to keep it simple rather than layering on ever more conditional compilation and runtime detections for each platform, opt-in works best. The current C reference doesn't have optimisations for windows, nor does PQClean.

Have added windows CI runners also for 32/64 bit x86 windows gnu/msvc as well as aarch64.

I've rewritten the at&t syntax GAS files into NASM, not tested on windows yet but it's passing all the test vectors on linux. Will keep it as an experimental feature for a while, it's behind the nasm feature gate on the nasm-conversion branch and will likely stay there for a while to be assessed:

https://github.com/Argyle-Software/kyber/tree/nasm-conversion