search

ninja-build / ninja

a small build system with a focus on speed
https://ninja-build.org/
Apache License 2.0
11.26k stars 1.6k forks source link

choking on lots of short paths (ninja: fatal: path has too many components) #1732

Closed maluke closed 10 months ago

maluke commented 4 years ago

ninja version 1.9.0 on win (msys2, mingw64)

The directories are 5 levels deep, but a lot of files are listed for implied dependencies on a rule.

Is ninja limiting the number of dependencies intentionally? Is something about my usage makes it thinks that's one big path? Is it just a bug?

$ ninja
ninja: fatal: path has too many components : lib/Cython/CodeWriter.py lib/Cython/Coverage.py lib/Cython/Debugging.py lib/Cython/Shadow.py lib/Cython/StringIOTree.py lib/Cython/Utils.py lib/Cython/__init__.py lib/Cython/Compiler/AnalysedTreeTransforms.py lib/Cython/Compiler/Annotate.py lib/Cython/Compiler/AutoDocTransforms.py lib/Cython/Compiler/Buffer.py lib/Cython/Compiler/Builtin.py lib/Cython/Compiler/CmdLine.py lib/Cython/Compiler/Code.pxd lib/Cython/Compiler/Code.py lib/Cython/Compiler/CodeGeneration.py lib/Cython/Compiler/CythonScope.py lib/Cython/Compiler/DebugFlags.py lib/Cython/Compiler/Errors.py lib/Cython/Compiler/ExprNodes.py lib/Cython/Compiler/FlowControl.pxd lib/Cython/Compiler/FlowControl.py lib/Cython/Compiler/FusedNode.py lib/Cython/Compiler/Future.py lib/Cython/Compiler/Interpreter.py lib/Cython/Compiler/Lexicon.py lib/Cython/Compiler/Main.py lib/Cython/Compiler/MemoryView.py lib/Cython/Compiler/ModuleNode.py lib/Cython/Compiler/Naming.py lib/Cython/Compiler/Nodes.py lib/Cython/Compiler/Optimize.py lib/Cython/Compiler/Options.py lib/Cython/Compiler/ParseTreeTransforms.pxd lib/Cython/Compiler/ParseTreeTransforms.py lib/Cython/Compiler/Parsing.pxd lib/Cython/Compiler/Parsing.py lib/Cython/Compiler/Pipeline.py lib/Cython/Compiler/PyrexTypes.py lib/Cython/Compiler/Pythran.py lib/Cython/Compiler/Scanning.pxd lib/Cython/Compiler/Scanning.py lib/Cython/Compiler/StringEncoding.py lib/Cython/Compiler/Symtab.py lib/Cython/Compiler/TreeFragment.py lib/Cython/Compiler/TreePath.py lib/Cython/Compiler/TypeInference.py lib/Cython/Compiler/TypeSlots.py lib/Cython/Compiler/UtilityCode.py lib/Cython/Compiler/UtilNodes.py lib/Cython/Compiler/Version.py lib/Cython/Compiler/Visitor.pxd lib/Cython/Compiler/Visitor.py lib/Cython/Compiler/__init__.py lib/Cython/Includes/openmp.pxd lib/Cython/Includes/cpython/array.pxd lib/Cython/Includes/cpython/bool.pxd lib/Cython/Includes/cpython/buffer.pxd lib/Cython/Includes/cpython/bytes.pxd lib/Cython/Includes/cpython/cobject.pxd lib/Cython/Includes/cpython/complex.pxd lib/Cython/Includes/cpython/datetime.pxd lib/Cython/Includes/cpython/dict.pxd lib/Cython/Includes/cpython/exc.pxd lib/Cython/Includes/cpython/float.pxd lib/Cython/Includes/cpython/function.pxd lib/Cython/Includes/cpython/getargs.pxd lib/Cython/Includes/cpython/instance.pxd lib/Cython/Includes/cpython/int.pxd lib/Cython/Includes/cpython/iterator.pxd lib/Cython/Includes/cpython/list.pxd lib/Cython/Includes/cpython/long.pxd lib/Cython/Includes/cpython/longintrepr.pxd lib/Cython/Includes/cpython/mapping.pxd lib/Cython/Includes/cpython/mem.pxd lib/Cython/Includes/cpython/method.pxd lib/Cython/Includes/cpython/module.pxd lib/Cython/Includes/cpython/number.pxd lib/Cython/Includes/cpython/object.pxd lib/Cython/Includes/cpython/oldbuffer.pxd lib/Cython/Includes/cpython/pycapsule.pxd lib/Cython/Includes/cpython/pystate.pxd lib/Cython/Includes/cpython/pythread.pxd lib/Cython/Includes/cpython/ref.pxd lib/Cython/Includes/cpython/sequence.pxd lib/Cython/Includes/cpython/set.pxd lib/Cython/Includes/cpython/slice.pxd lib/Cython/Includes/cpython/string.pxd lib/Cython/Includes/cpython/tuple.pxd lib/Cython/Includes/cpython/type.pxd lib/Cython/Includes/cpython/unicode.pxd lib/Cython/Includes/cpython/version.pxd lib/Cython/Includes/cpython/weakref.pxd lib/Cython/Includes/cpython/__init__.pxd lib/Cython/Includes/libc/errno.pxd lib/Cython/Includes/libc/float.pxd lib/Cython/Includes/libc/limits.pxd lib/Cython/Includes/libc/locale.pxd lib/Cython/Includes/libc/math.pxd lib/Cython/Includes/libc/setjmp.pxd lib/Cython/Includes/libc/signal.pxd lib/Cython/Includes/libc/stddef.pxd lib/Cython/Includes/libc/stdint.pxd lib/Cython/Includes/libc/stdio.pxd lib/Cython/Includes/libc/stdlib.pxd lib/Cython/Includes/libc/string.pxd lib/Cython/Includes/libc/time.pxd lib/Cython/Includes/libc/__init__.pxd lib/Cython/Includes/libcpp/algorithm.pxd lib/Cython/Includes/libcpp/cast.pxd lib/Cython/Includes/libcpp/complex.pxd lib/Cython/Includes/libcpp/deque.pxd lib/Cython/Includes/libcpp/functional.pxd lib/Cython/Includes/libcpp/iterator.pxd lib/Cython/Includes/libcpp/limits.pxd lib/Cython/Includes/libcpp/list.pxd lib/Cython/Includes/libcpp/map.pxd lib/Cython/Includes/libcpp/memory.pxd lib/Cython/Includes/libcpp/pair.pxd lib/Cython/Includes/libcpp/queue.pxd lib/Cython/Includes/libcpp/set.pxd lib/Cython/Includes/libcpp/stack.pxd lib/Cython/Includes/libcpp/string.pxd lib/Cython/Includes/libcpp/typeindex.pxd lib/Cython/Includes/libcpp/typeinfo.pxd lib/Cython/Includes/libcpp/unordered_map.pxd lib/Cython/Includes/libcpp/unordered_set.pxd lib/Cython/Includes/libcpp/utility.pxd lib/Cython/Includes/libcpp/vector.pxd lib/Cython/Includes/libcpp/__init__.pxd lib/Cython/Includes/numpy/math.pxd lib/Cython/Includes/numpy/__init__.pxd lib/Cython/Includes/posix/dlfcn.pxd lib/Cython/Includes/posix/fcntl.pxd lib/Cython/Includes/posix/ioctl.pxd lib/Cython/Includes/posix/mman.pxd lib/Cython/Includes/posix/resource.pxd lib/Cython/Includes/posix/select.pxd lib/Cython/Includes/posix/signal.pxd lib/Cython/Includes/posix/stat.pxd lib/Cython/Includes/posix/stdio.pxd lib/Cython/Includes/posix/stdlib.pxd lib/Cython/Includes/posix/strings.pxd lib/Cython/Includes/posix/time.pxd lib/Cython/Includes/posix/types.pxd lib/Cython/Includes/posix/unistd.pxd lib/Cython/Includes/posix/wait.pxd lib/Cython/Includes/posix/__init__.pxd lib/Cython/Parser/ConcreteSyntaxTree.pyx lib/Cython/Parser/Grammar lib/Cython/Parser/__init__.py lib/Cython/Plex/Actions.py lib/Cython/Plex/DFA.py lib/Cython/Plex/Errors.py lib/Cython/Plex/Lexicons.py lib/Cython/Plex/Machines.py lib/Cython/Plex/Regexps.py lib/Cython/Plex/Scanners.py lib/Cython/Plex/Timing.py lib/Cython/Plex/Traditional.py lib/Cython/Plex/Transitions.py lib/Cython/Plex/__init__.py lib/Cython/Tempita/compat3.py lib/Cython/Tempita/token.py lib/Cython/Tempita/tokenize.py lib/Cython/Tempita/_looper.py lib/Cython/Tempita/_tempita.py lib/Cython/Tempita/__init__.py lib/Cython/Utility/arrayarray.h lib/Cython/Utility/AsyncGen.c lib/Cython/Utility/Buffer.c lib/Cython/Utility/Builtins.c lib/Cython/Utility/Capsule.c lib/Cython/Utility/CConvert.pyx lib/Cython/Utility/CMath.c lib/Cython/Utility/CommonStructures.c lib/Cython/Utility/Coroutine.c lib/Cython/Utility/CpdefEnums.pyx lib/Cython/Utility/CppConvert.pyx lib/Cython/Utility/CppSupport.cpp lib/Cython/Utility/CythonFunction.c lib/Cython/Utility/Embed.c lib/Cython/Utility/Exceptions.c lib/Cython/Utility/ExtensionTypes.c lib/Cython/Utility/FunctionArguments.c lib/Cython/Utility/ImportExport.c lib/Cython/Utility/MemoryView.pyx lib/Cython/Utility/MemoryView_C.c lib/Cython/Utility/ModuleSetupCode.c lib/Cython/Utility/ObjectHandling.c lib/Cython/Utility/Optimize.c lib/Cython/Utility/Overflow.c lib/Cython/Utility/Printing.c lib/Cython/Utility/Profile.c lib/Cython/Utility/StringTools.c lib/Cython/Utility/TestCythonScope.pyx lib/Cython/Utility/TestCyUtilityLoader.pyx lib/Cython/Utility/TestUtilityLoader.c lib/Cython/Utility/TypeConversion.c lib/Cython/Utility/__init__.py

build.ninja:

cython-deps = lib/Cython/CodeWriter.py lib/Cython/Coverage.py lib/Cython/Debugging.py lib/Cython/Shadow.py lib/Cython/StringIOTree.py lib/Cython/Utils.py lib/Cython/__init__.py lib/Cython/Compiler/AnalysedTreeTransforms.py lib/Cython/Compiler/Annotate.py lib/Cython/Compiler/AutoDocTransforms.py lib/Cython/Compiler/Buffer.py lib/Cython/Compiler/Builtin.py lib/Cython/Compiler/CmdLine.py lib/Cython/Compiler/Code.pxd lib/Cython/Compiler/Code.py lib/Cython/Compiler/CodeGeneration.py lib/Cython/Compiler/CythonScope.py lib/Cython/Compiler/DebugFlags.py lib/Cython/Compiler/Errors.py lib/Cython/Compiler/ExprNodes.py lib/Cython/Compiler/FlowControl.pxd lib/Cython/Compiler/FlowControl.py lib/Cython/Compiler/FusedNode.py lib/Cython/Compiler/Future.py lib/Cython/Compiler/Interpreter.py lib/Cython/Compiler/Lexicon.py lib/Cython/Compiler/Main.py lib/Cython/Compiler/MemoryView.py lib/Cython/Compiler/ModuleNode.py lib/Cython/Compiler/Naming.py lib/Cython/Compiler/Nodes.py lib/Cython/Compiler/Optimize.py lib/Cython/Compiler/Options.py lib/Cython/Compiler/ParseTreeTransforms.pxd lib/Cython/Compiler/ParseTreeTransforms.py lib/Cython/Compiler/Parsing.pxd lib/Cython/Compiler/Parsing.py lib/Cython/Compiler/Pipeline.py lib/Cython/Compiler/PyrexTypes.py lib/Cython/Compiler/Pythran.py lib/Cython/Compiler/Scanning.pxd lib/Cython/Compiler/Scanning.py lib/Cython/Compiler/StringEncoding.py lib/Cython/Compiler/Symtab.py lib/Cython/Compiler/TreeFragment.py lib/Cython/Compiler/TreePath.py lib/Cython/Compiler/TypeInference.py lib/Cython/Compiler/TypeSlots.py lib/Cython/Compiler/UtilityCode.py lib/Cython/Compiler/UtilNodes.py lib/Cython/Compiler/Version.py lib/Cython/Compiler/Visitor.pxd lib/Cython/Compiler/Visitor.py lib/Cython/Compiler/__init__.py lib/Cython/Includes/openmp.pxd lib/Cython/Includes/cpython/array.pxd lib/Cython/Includes/cpython/bool.pxd lib/Cython/Includes/cpython/buffer.pxd lib/Cython/Includes/cpython/bytes.pxd lib/Cython/Includes/cpython/cobject.pxd lib/Cython/Includes/cpython/complex.pxd lib/Cython/Includes/cpython/datetime.pxd lib/Cython/Includes/cpython/dict.pxd lib/Cython/Includes/cpython/exc.pxd lib/Cython/Includes/cpython/float.pxd lib/Cython/Includes/cpython/function.pxd lib/Cython/Includes/cpython/getargs.pxd lib/Cython/Includes/cpython/instance.pxd lib/Cython/Includes/cpython/int.pxd lib/Cython/Includes/cpython/iterator.pxd lib/Cython/Includes/cpython/list.pxd lib/Cython/Includes/cpython/long.pxd lib/Cython/Includes/cpython/longintrepr.pxd lib/Cython/Includes/cpython/mapping.pxd lib/Cython/Includes/cpython/mem.pxd lib/Cython/Includes/cpython/method.pxd lib/Cython/Includes/cpython/module.pxd lib/Cython/Includes/cpython/number.pxd lib/Cython/Includes/cpython/object.pxd lib/Cython/Includes/cpython/oldbuffer.pxd lib/Cython/Includes/cpython/pycapsule.pxd lib/Cython/Includes/cpython/pystate.pxd lib/Cython/Includes/cpython/pythread.pxd lib/Cython/Includes/cpython/ref.pxd lib/Cython/Includes/cpython/sequence.pxd lib/Cython/Includes/cpython/set.pxd lib/Cython/Includes/cpython/slice.pxd lib/Cython/Includes/cpython/string.pxd lib/Cython/Includes/cpython/tuple.pxd lib/Cython/Includes/cpython/type.pxd lib/Cython/Includes/cpython/unicode.pxd lib/Cython/Includes/cpython/version.pxd lib/Cython/Includes/cpython/weakref.pxd lib/Cython/Includes/cpython/__init__.pxd lib/Cython/Includes/libc/errno.pxd lib/Cython/Includes/libc/float.pxd lib/Cython/Includes/libc/limits.pxd lib/Cython/Includes/libc/locale.pxd lib/Cython/Includes/libc/math.pxd lib/Cython/Includes/libc/setjmp.pxd lib/Cython/Includes/libc/signal.pxd lib/Cython/Includes/libc/stddef.pxd lib/Cython/Includes/libc/stdint.pxd lib/Cython/Includes/libc/stdio.pxd lib/Cython/Includes/libc/stdlib.pxd lib/Cython/Includes/libc/string.pxd lib/Cython/Includes/libc/time.pxd lib/Cython/Includes/libc/__init__.pxd lib/Cython/Includes/libcpp/algorithm.pxd lib/Cython/Includes/libcpp/cast.pxd lib/Cython/Includes/libcpp/complex.pxd lib/Cython/Includes/libcpp/deque.pxd lib/Cython/Includes/libcpp/functional.pxd lib/Cython/Includes/libcpp/iterator.pxd lib/Cython/Includes/libcpp/limits.pxd lib/Cython/Includes/libcpp/list.pxd lib/Cython/Includes/libcpp/map.pxd lib/Cython/Includes/libcpp/memory.pxd lib/Cython/Includes/libcpp/pair.pxd lib/Cython/Includes/libcpp/queue.pxd lib/Cython/Includes/libcpp/set.pxd lib/Cython/Includes/libcpp/stack.pxd lib/Cython/Includes/libcpp/string.pxd lib/Cython/Includes/libcpp/typeindex.pxd lib/Cython/Includes/libcpp/typeinfo.pxd lib/Cython/Includes/libcpp/unordered_map.pxd lib/Cython/Includes/libcpp/unordered_set.pxd lib/Cython/Includes/libcpp/utility.pxd lib/Cython/Includes/libcpp/vector.pxd lib/Cython/Includes/libcpp/__init__.pxd lib/Cython/Includes/numpy/math.pxd lib/Cython/Includes/numpy/__init__.pxd lib/Cython/Includes/posix/dlfcn.pxd lib/Cython/Includes/posix/fcntl.pxd lib/Cython/Includes/posix/ioctl.pxd lib/Cython/Includes/posix/mman.pxd lib/Cython/Includes/posix/resource.pxd lib/Cython/Includes/posix/select.pxd lib/Cython/Includes/posix/signal.pxd lib/Cython/Includes/posix/stat.pxd lib/Cython/Includes/posix/stdio.pxd lib/Cython/Includes/posix/stdlib.pxd lib/Cython/Includes/posix/strings.pxd lib/Cython/Includes/posix/time.pxd lib/Cython/Includes/posix/types.pxd lib/Cython/Includes/posix/unistd.pxd lib/Cython/Includes/posix/wait.pxd lib/Cython/Includes/posix/__init__.pxd lib/Cython/Parser/ConcreteSyntaxTree.pyx lib/Cython/Parser/Grammar lib/Cython/Parser/__init__.py lib/Cython/Plex/Actions.py lib/Cython/Plex/DFA.py lib/Cython/Plex/Errors.py lib/Cython/Plex/Lexicons.py lib/Cython/Plex/Machines.py lib/Cython/Plex/Regexps.py lib/Cython/Plex/Scanners.py lib/Cython/Plex/Timing.py lib/Cython/Plex/Traditional.py lib/Cython/Plex/Transitions.py lib/Cython/Plex/__init__.py lib/Cython/Tempita/compat3.py lib/Cython/Tempita/token.py lib/Cython/Tempita/tokenize.py lib/Cython/Tempita/_looper.py lib/Cython/Tempita/_tempita.py lib/Cython/Tempita/__init__.py lib/Cython/Utility/arrayarray.h lib/Cython/Utility/AsyncGen.c lib/Cython/Utility/Buffer.c lib/Cython/Utility/Builtins.c lib/Cython/Utility/Capsule.c lib/Cython/Utility/CConvert.pyx lib/Cython/Utility/CMath.c lib/Cython/Utility/CommonStructures.c lib/Cython/Utility/Coroutine.c lib/Cython/Utility/CpdefEnums.pyx lib/Cython/Utility/CppConvert.pyx lib/Cython/Utility/CppSupport.cpp lib/Cython/Utility/CythonFunction.c lib/Cython/Utility/Embed.c lib/Cython/Utility/Exceptions.c lib/Cython/Utility/ExtensionTypes.c lib/Cython/Utility/FunctionArguments.c lib/Cython/Utility/ImportExport.c lib/Cython/Utility/MemoryView.pyx lib/Cython/Utility/MemoryView_C.c lib/Cython/Utility/ModuleSetupCode.c lib/Cython/Utility/ObjectHandling.c lib/Cython/Utility/Optimize.c lib/Cython/Utility/Overflow.c lib/Cython/Utility/Printing.c lib/Cython/Utility/Profile.c lib/Cython/Utility/StringTools.c lib/Cython/Utility/TestCythonScope.pyx lib/Cython/Utility/TestCyUtilityLoader.pyx lib/Cython/Utility/TestUtilityLoader.c lib/Cython/Utility/TypeConversion.c lib/Cython/Utility/__init__.py

build src/mod/_pyx.c: cython src/mod/_pyx.pyx | $cython-deps
jhasse commented 4 years ago

See #1161.

jhasse commented 4 years ago

Maybe we could fallback to a "slow" version of CanonicalizePath when the maximum is reached (i.e. one with a heap allocated array).

constructor-s commented 4 years ago

Also having what appears to be the same/similar issue in a Visual Studio 2019 CMake project once I link against Qt5 (target_link_libraries(CMakeProjectHelloWorld Qt5::Widgets)):

ninja: fatal: path has too many components : E:\vcpkg\installed\x64-windows-static\debug\lib\Qt5Cored.lib "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\mpr.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\netapi32.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\userenv.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\version.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\ws2_32.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\kernel32.lib" "E:\Windows Kits\10\lib\10.0.18362.0\um\x64\user32.lib" "E:\Windows Kits\10\lib\10.0.18362.0\user32.lib" "E:\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64\\shell32.lib" "E:\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64\\uuid.lib" "E:\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64\\ole32.lib" "E:\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64\\advapi32.lib" "E:\\Windows Kits\\10\\lib\\10.0.18362.0\\um\\x64\\winmm.lib"

The ninja.exe bundled with Visual Studio at E:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe was 1.8.2 but even if I replaced this with 1.10.0 I am encountering the same issue.

KUGA2 commented 1 year ago

I have just hit this limit. On Linux.

I did a script counting the slashes, and the issue is not reaching the path separator limit (as in #1161 or #1181). My maximum is 17 slashes.

However the "path" is more then 10MB long and has 133782 paths (semicolons). Strange this is that this happens on a build server in a build container on one server, but not on others.

edit: I found the issue in my CMake, the dependency list of a target was very long (wrong glob). But nevertheless the error is very hard to fix.

mbratch commented 11 months ago

I just ran into this limit on Windows using CMake 3.14.5 and ninja. Number of files in path: 33 Length of path in bytes: 1049089