Closed DrTimothyAldenDavis closed 2 weeks ago
-1073740791 is 0xc0000409 in hexadecimal. 0xc0000409 means STATUS_STACK_BUFFER_OVERRUN.
I'm guessing that error is from running the JIT package generator executable (not actually from cmd.exe). That could be caused by an array index that is too large (positive or negative) when indexing a variable that is allocated on the stack. Maybe an erroneous integer typecast? I wonder why that doesn't happen in the CI here though.
Which cmake command do you use to configure GRAPHBLAS?
Is the problem visible in a Debug build? cmake -DCMAKE_BUILD_TYPE=Debug ....
I know MSVC's C++ Debug builds have bounds checks on iterators, maybe they also have checks that might help identify this. If not, maybe run in a debugger?
Other differences to the CI are Visual Studio 2019 vs Visual Studio 2022 and Visual Studio vs. Ninja as the CMake generator. But that shouldn't matter. Or could it?
@alugowski: Unsure if this is related: UNIX Makefiles are sometimes a bit iffy on Windows. Do you have the option to switch to the Ninja generator in your CI rules? That is working better on Windows in my experience. (It is working fine on any platform where I tried it.)
@alugowski: Unsure if this is related: UNIX Makefiles are sometimes a bit iffy on Windows. Do you have the option to switch to the Ninja generator in your CI rules? That is working better on Windows in my experience. (It is working fine on any platform where I tried it.)
Don't need to, Makefiles works in all cases we need. I remember I had to explicitly add that argument to solve an issue that I've forgotten, but I believe it was actually to fix something on Windows.
My preference is to use Ninja whenever possible as it's nicer on Unix too, but alas that wasn't an option. Note the official instructions only use make
, no reference to the standard cmake commands. My preference is to stick to official instructions if possible, leave the defaults unchanged if possible, and only change settings if forced. Yes that Python repo is full of hacks, but each one was mandatory at some point and most probably still are.
I tried to reproduce the error with a current head of the dev2
branch of the SuiteSparse repository (3df9cb323101e7a5b6b84ca9ee04a27e42e70068). Is that in sync with the current GraphBLAS repository?
I used the following commands (in .../SuiteSparse/GraphBLAS/build
) using the "x64 Native Tools Command Prompt for VS 2022" (that is the "Community" edition) on Windows 11 23H2:
cmake ..
cmake --build . --config Release
That also picked the Visual Studio generator. But it is building without issues here. In particular, the JIT package generator was built and it generated the sources without issues. After that, building the object files continued without any errors or warnings.
Edit: Result of the above configure step:
>cmake ..
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
-- The C compiler identification is MSVC 19.40.33808.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Building SuiteSparse:GraphBLAS version: v9.3.0, date: June 1, 2024
-- GraphBLAS C API: v2.1, date: Dec 22, 2023
-- Source: D:/repo/SuiteSparse/SuiteSparse/GraphBLAS
-- Build: D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/build
-- Install lib: lib
-- Install include: include/suitesparse
-- Install bin: bin
-- Install pkg-file: lib
-- Install rpath: $ORIGIN
-- Build rpath:
-- Build type: Release
-- Looking for a Fortran compiler
-- Looking for a Fortran compiler - C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe
-- The Fortran compiler identification is Intel 2021.0.0.20240222
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Determine Intel Fortran Compiler Implicit Link Path
-- Determine Intel Fortran Compiler Implicit Link Path - done
-- Check for working Fortran compiler: C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe - skipped
-- Fortran: C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- CUDA: not found
-- CUDA: not enabled
-- GraphBLAS CUDA JIT: disabled
-- GraphBLAS CPU JIT: enabled
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0") found components: C
-- GraphBLAS has OpenMP: ON
-- cpu_features (by google.com): enabled
-- Performing Test GxB_HAVE_COMPLEX_C99
-- Performing Test GxB_HAVE_COMPLEX_C99 - Failed
-- Performing Test GxB_HAVE_COMPLEX_MSVC
-- Performing Test GxB_HAVE_COMPLEX_MSVC - Success
-- Building dynamic GraphBLAS library (no static)
-- Looking for fmax
-- Looking for fmax - found
-- Performing Test TEST_FOR_STDATOMIC
-- Performing Test TEST_FOR_STDATOMIC - Success
-- C11 atomics: OK. -latomic not needed
-- CMAKE OpenMP libraries:
-- CMAKE OpenMP include:
-- CMAKE OpenMP C flags: -openmp
-- CMAKE C flags: /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG
-- Skipping the demos in GraphBLAS/Demo
-- ------------------------------------------------------------------------
-- JIT configuration:
-- ------------------------------------------------------------------------
-- JIT C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
-- JIT C flags: /DWIN32 /D_WINDOWS /O2 -wd\"4244\" -wd\"4146\" -wd\"4018\" -wd\"4996\" -wd\"4047\" -wd\"4554\" /O2 /Ob2 /DNDEBUG -openmp
-- JIT link flags: /machine:x64
-- JIT lib prefix:
-- JIT lib suffix: .dll
-- JIT obj suffix: .obj
-- JIT cache: C:\Users\Markus\AppData\Local/SuiteSparse/GrB9.3.0
-- JIT openmp inc:
-- JIT openmp dirs
-- JIT libraries:
-- JIT cmake libs:
-- ------------------------------------------------------------------------
-- CMAKE report for: GraphBLAS
-- ------------------------------------------------------------------------
-- inside common SuiteSparse root: OFF
-- install in SuiteSparse/lib and SuiteSparse/include: OFF
-- build type: Release
-- BUILD_SHARED_LIBS: ON
-- BUILD_STATIC_LIBS: OFF
-- C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
-- C flags: /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG
-- C Flags release: /O2 /Ob2 /DNDEBUG
-- compile definitions: _CRT_SECURE_NO_WARNINGS
-- ------------------------------------------------------------------------
-- Configuring done (83.9s)
-- Generating done (0.2s)
-- Build files have been written to: D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/build
Relevant part of the build step:
>cmake --build . --config Release
MSBuild-Version 17.10.4+10fbfbf2e für .NET Framework
1>Checking Build System
Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/JITpackage/CMakeLists.txt
grb_jitpackage.c
grb_jitpackage.vcxproj -> D:\repo\SuiteSparse\SuiteSparse\GraphBLAS\build\JITpackage\Release\grb_jitpackage.exe
Generating compressed sources for JIT compiler...
Processing 242 input files ...
Total uncompressed: 1909807 bytes
Total compressed: 330640 bytes
Compression: 0.173127
Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/JITpackage/CMakeLists.txt
Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/CMakeLists.txt
GB_JITpackage.c
GB_prejit.c
GB_all_aliased.c
GB_any_aliased.c
GB_is_shallow.c
GB_apply.c
[...]
Fwiw, the generator executable only links to the C runtime and the kernel for me:
$ ntldd grb_jitpackage.exe
VCRUNTIME140.dll => C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (0x0000015518ad0000)
KERNEL32.dll => C:\WINDOWS\SYSTEM32\KERNEL32.dll (0x0000015518af0000)
Or with Visual Studio tools:
...\SuiteSparse\GraphBLAS\build>dumpbin /dependents .\JITpackage\Release\grb_jitpackage.exe
Microsoft (R) COFF/PE Dumper Version 14.40.33808.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file .\JITpackage\Release\grb_jitpackage.exe
File Type: EXECUTABLE IMAGE
Image has the following dependencies:
VCRUNTIME140.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
KERNEL32.dll
Summary
1000 .data
2000 .pdata
5000 .rdata
1000 .reloc
1000 .rsrc
50000 .text
Just another idea: The maximum number of characters for a command line in cmd.exe
is 8191 (in some Windows versions):
https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation
The sources are checked out at a deeper absolute path for you than they are for me. A glob command is used to generate a string that contains the absolute paths to all relevant sources files. That string is passed as an argument list to the JIT package generator.
It might be that the longer base path on your checkout location leads to the command line length limit being exceeded.
Does it work if you checkout the GraphBLAS sources to a path that is shorter (e.g., to C:\repo\GraphBLAS
)?
If it does, a potential solution could be to write the list of relevant source files to a text file and pass that text file as an argument to the JIT package generator. The JIT package generator would need to be changed to read the file list from that text file instead of iterating through its command line arguments.
I opened #302 to address the potential issue described in the previous comment.
Just another idea: The maximum number of characters for a command line in cmd.exe is 8191: https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation
Oh -- that's got to be it. There are 243 or so files in the JITpackage, and I'm working in a folder that is C: ... something here ... Documents/Github/GraphBLAS/build ...
Leave it MS to report error MSB6006: "cmd.exe" exited with code -1073740791
and not "error: command line too long ...". Sigh.
The 'something here' is my Windows home folder (I'm not on my Windows machine at the moment). I think that itself might be deeper than normal because it's a departmentally-managed machine. I don't have admin access, so I can't do C:/my/GraphBLAS. Adding to the complexity is that I so rarely use Windows at all, for anything, let alone development.
I will try to shorten the paths of my files to see if that works. I will rewrite the JITpackage to write out the set of filenames to a file, in the cmake build process.
Yes, the dev2 branch of GraphBLAS and the dev2 branch of SuiteSparse are currently identical. I also tend to keep the default branches of the 2 repos synced.
When I started GraphBLAS, I didn't place it inside my SuiteSparse repo because it was a stand-alone project, but then I decided later it should be part of SuiteSparse. But by then, lots of people were getting GraphBLAS from its own repo, not inside SuiteSparse. Thus the two repos. I guess eventually I should close down the stand-alone GraphBLAS repo and use SuiteSparse as a whole, instead, since it's more comprehensive (building GraphBLAS from the root CMakeLists.txt in SuiteSparse is very handy on Windows). Plus the github CI is far better.
I will rewrite the JITpackage to write out the set of filenames to a file, in the cmake build process.
Oh you beat me to it :-) ... https://github.com/DrTimothyAldenDavis/GraphBLAS/pull/302 ... thanks!
Fixed with #302 and subsequent updates.
When I try to build GraphBLAS in its own CMakeLists.txt build file (not as part of the root SuiteSparse build), I get this error. I'm using cmake 3.26 and am building the GraphBLAS project in Visual Studio:
I can't tell how the grb_jitpackage.exe is failing. I do see a grb_jitpackage.exe file that is compiled where I expect to see it ( Documents/GitHub/GraphBLAS/build/JITpackage/Release/grb_jitpackage.exe ).
My cmake configure output looks OK:
In the SuiteSparse CI, the build looks different, and it succeeds, when using the SuiteSparse root CMakeLists.txt:
@mmuetzel : any ideas?
@alugowski is seeing a similar problem here: https://github.com/GraphBLAS/python-suitesparse-graphblas/pull/128 , see https://github.com/GraphBLAS/python-suitesparse-graphblas/actions/runs/9343575516/job/25713378326#step:9:351