search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.53k stars 1.19k forks source link

[BUG] Setuptools passes too long a command to link.exe, causing [WinError 206] The filename or extension is too long #4177

Open Mause opened 10 months ago

Mause commented 10 months ago

setuptools version

setuptools==69.0.3

Python version

Python 3.12

OS

Windows

Additional environment information

This is occuring on Windows build agents on Github Actions, when we updated to start building for Python 3.12 (and hence had to switch to using setuptools' distutils fork). Previously we had SETUPTOOLS_USE_DISTUTILS=stdlib set due to an old numpy issue.

This issue is related in the error code to https://github.com/pypa/setuptools/issues/3591, but the issue there doesn't seem to be the same

Description

When we switch from SETUPTOOLS_USE_DISTUTILS=stdlib to the default of local, we started seeing [WinError 206] The filename or extension is too long (though we had to patch distutils to stop it from swallowing this exception and only returning failed: None)

After further investigation, we found that the call to link.exe was longer than the 32766 characters permitted by Windows, hence causing the above error.

For now, we've worked around this by patching distutils to call GetShortPathNameW, which manages to shorten the call by around 12000 characters

Expected behavior

The call to link.exe should be shorter than the maximum allowed by Windows, and compilation should succeed, as it did with the version of distutils includes in < Python 3.12

How to Reproduce

  1. Clone https://github.com/duckdb/duckdb @ e37e8eecc04a7e904ab0ca93e76595d25f02217e
  2. Run pip install -e . in tools/pythonpkg on Windows 10+

Output

Command in question:

Details

```console "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\runneradmin\AppData\Local\Temp\cibw-run-515uywyt\cp312-win_amd64\build\venv\libs /LIBPATH:C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.0\tools\libs /LIBPATH:C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.0\tools /LIBPATH:C:\Users\runneradmin\AppData\Local\Temp\cibw-run-515uywyt\cp312-win_amd64\build\venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\um\x64" /EXPORT:PyInit_duckdb build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\fts\fts_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\fts\fts_indexing.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-dateadd.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-datefunc.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-datepart.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-datesub.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-datetrunc.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-list-range.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-makedate.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-strptime.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-table-range.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-timebucket.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu-timezone.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\.\icu_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\icu\third_party\icu\stubdata\stubdata.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\buffered_json_reader.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_common.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_deserializer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_enums.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_functions.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_scan.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\json_serializer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\serialize_json.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\json\yyjson\yyjson.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\column_reader.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\column_writer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_crypto.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_metadata.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_reader.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_statistics.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_timestamp.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\parquet_writer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\serialize_parquet.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\parquet\zstd_file_system.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\bm_utils.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\build.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\dbgen.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\dbgen_gunk.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\permute.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\rnd.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\rng64.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\speed_seed.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\dbgen\text.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\tpch\tpch_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\boolean_operators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\comparison_operators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\generators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\is_distinct_from.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\null_operations.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\numeric_inplace_operators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\vector_cast.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\vector_copy.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\vector_hash.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\common\vector_operations\vector_storage.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\copied_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\deserialized_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\external_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\no_operator_caching_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\parsed_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\prepared_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\src\verification\unoptimized_statement_verifier.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\fastpforlib\bitpacking.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\fmt\format.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\fsst\fsst_avx512.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\fsst\libfsst.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\hyperloglog\hyperloglog.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\hyperloglog\sds.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\pg_functions.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\postgres_parser.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_nodes_list.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_nodes_makefuncs.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_nodes_value.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_parser_gram.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_parser_parser.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_parser_scan.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_backend_parser_scansup.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\libpg_query\src_common_keywords.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\aes.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\aria.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\asn1parse.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\base64.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\bignum.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\camellia.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\cipher.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\cipher_wrap.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\constant_time.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\entropy.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\entropy_poll.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\gcm.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\md.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\oid.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\pem.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\pk.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\pk_wrap.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\pkparse.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\platform_util.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\rsa.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\rsa_alt_helpers.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\sha1.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\sha256.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\library\sha512.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\mbedtls\mbedtls_wrapper.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\miniz\miniz.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\parquet\parquet_constants.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\parquet\parquet_types.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\bitstate.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\compile.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\dfa.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\filtered_re2.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\mimics_pcre.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\nfa.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\onepass.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\parse.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\perl_groups.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\prefilter.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\prefilter_tree.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\prog.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\re2.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\regexp.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\set.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\simplify.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\stringpiece.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\tostring.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\unicode_casefold.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\re2\unicode_groups.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\util\rune.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\re2\util\strutil.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\skiplist\SkipList.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snappy\snappy-sinksource.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snappy\snappy.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\libstemmer\libstemmer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\runtime\api.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\runtime\utilities.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_arabic.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_basque.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_catalan.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_danish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_dutch.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_english.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_finnish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_french.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_german.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_german2.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_greek.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_hindi.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_hungarian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_indonesian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_irish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_italian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_kraaij_pohlmann.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_lithuanian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_lovins.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_nepali.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_norwegian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_porter.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_portuguese.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_romanian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_russian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_serbian.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_spanish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_swedish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_tamil.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\snowball\src_c\stem_UTF_8_turkish.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\thrift\thrift\protocol\TProtocol.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\thrift\thrift\transport\TBufferTransports.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\thrift\thrift\transport\TTransportException.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\utf8proc\utf8proc.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\utf8proc\utf8proc_wrapper.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\common\entropy_common.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\common\error_private.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\common\fse_decompress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\common\xxhash.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\common\zstd_common.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\fse_compress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\hist.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\huf_compress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_compress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_compress_literals.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_compress_sequences.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_compress_superblock.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_double_fast.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_fast.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_lazy.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_ldm.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\compress\zstd_opt.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\decompress\huf_decompress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\decompress\zstd_ddict.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\decompress\zstd_decompress.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\zstd\decompress\zstd_decompress_block.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_extension_icu_third_party_icu_common.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_extension_icu_third_party_icu_i18n.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_extension_json_json_functions.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_catalog.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_catalog_catalog_entry.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_catalog_catalog_entry_dependency.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_catalog_default.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_adbc.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_adbc_nanoarrow.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_arrow.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_arrow_appender.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_crypto.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_enums.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_operator.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_progress_bar.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_row_operations.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_serializer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_sort.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_types.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_types_column.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_types_row.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_common_value_operations.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_aggregate_algebraic.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_aggregate_distributive.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_aggregate_holistic.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_aggregate_nested.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_aggregate_regression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_array.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_bit.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_blob.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_date.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_debug.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_enum.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_generic.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_list.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_map.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_math.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_operators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_random.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_secret.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_string.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_struct.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_core_functions_scalar_union.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_expression_executor.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_index.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_index_art.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_nested_loop_join.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_aggregate.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_csv_scanner.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_csv_scanner_sniffer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_filter.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_helper.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_join.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_order.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_persistent.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_projection.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_scan.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_schema.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_operator_set.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_execution_physical_plan.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_aggregate.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_aggregate_distributive.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_cast.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_cast_union.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_pragma.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_compressed_materialization.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_generic.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_list.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_operators.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_sequence.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_string.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_string_regexp.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_struct.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_scalar_system.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_table.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_table_arrow.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_table_system.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_function_table_version.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_capi.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_capi_cast.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_chunk_scan_state.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_extension.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_relation.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_secret.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_main_settings.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_compressed_materialization.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_join_order.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_matcher.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_pullup.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_pushdown.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_rule.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_statistics_expression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_optimizer_statistics_operator.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parallel.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_constraints.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_expression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_parsed_data.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_query_node.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_statement.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_tableref.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_transform_constraint.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_transform_expression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_transform_helpers.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_transform_statement.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_parser_transform_tableref.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_binder_expression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_binder_query_node.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_binder_statement.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_binder_tableref.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_expression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_expression_binder.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_filter.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_operator.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_planner_subquery.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_buffer.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_checkpoint.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_compression.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_compression_chimp.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_metadata.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_serialization.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_statistics.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_storage_table.obj build\temp.win-amd64-cpython-312\Release\duckdb_build\ub_src_transaction.obj build\temp.win-amd64-cpython-312\Release\duckdb_python.obj build\temp.win-amd64-cpython-312\Release\src\arrow\arrow_array_stream.obj build\temp.win-amd64-cpython-312\Release\src\arrow\arrow_export_utils.obj build\temp.win-amd64-cpython-312\Release\src\common\exceptions.obj build\temp.win-amd64-cpython-312\Release\src\dataframe.obj build\temp.win-amd64-cpython-312\Release\src\functional\functional.obj build\temp.win-amd64-cpython-312\Release\src\importer.obj build\temp.win-amd64-cpython-312\Release\src\jupyter\jupyter_progress_bar_display.obj build\temp.win-amd64-cpython-312\Release\src\map.obj build\temp.win-amd64-cpython-312\Release\src\native\python_conversion.obj build\temp.win-amd64-cpython-312\Release\src\native\python_objects.obj build\temp.win-amd64-cpython-312\Release\src\numpy\array_wrapper.obj build\temp.win-amd64-cpython-312\Release\src\numpy\numpy_bind.obj build\temp.win-amd64-cpython-312\Release\src\numpy\numpy_result_conversion.obj build\temp.win-amd64-cpython-312\Release\src\numpy\numpy_scan.obj build\temp.win-amd64-cpython-312\Release\src\numpy\raw_array_wrapper.obj build\temp.win-amd64-cpython-312\Release\src\numpy\type.obj build\temp.win-amd64-cpython-312\Release\src\pandas\analyzer.obj build\temp.win-amd64-cpython-312\Release\src\pandas\bind.obj build\temp.win-amd64-cpython-312\Release\src\pandas\scan.obj build\temp.win-amd64-cpython-312\Release\src\path_like.obj build\temp.win-amd64-cpython-312\Release\src\pybind11\pybind_wrapper.obj build\temp.win-amd64-cpython-312\Release\src\pyconnection.obj build\temp.win-amd64-cpython-312\Release\src\pyconnection\type_creation.obj build\temp.win-amd64-cpython-312\Release\src\pyduckdb\connection_wrapper.obj build\temp.win-amd64-cpython-312\Release\src\pyexpression.obj build\temp.win-amd64-cpython-312\Release\src\pyexpression\initialize.obj build\temp.win-amd64-cpython-312\Release\src\pyfilesystem.obj build\temp.win-amd64-cpython-312\Release\src\pyrelation.obj build\temp.win-amd64-cpython-312\Release\src\pyrelation\initialize.obj build\temp.win-amd64-cpython-312\Release\src\pyresult.obj build\temp.win-amd64-cpython-312\Release\src\python_import_cache.obj build\temp.win-amd64-cpython-312\Release\src\python_udf.obj build\temp.win-amd64-cpython-312\Release\src\typing\pytype.obj build\temp.win-amd64-cpython-312\Release\src\typing\typing.obj /OUT:build\lib.win-amd64-cpython-312\duckdb\duckdb.cp312-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-312\Release\duckdb_build\extension\fts\duckdb.cp312-win_amd64.lib /wd4244 /wd4267 /wd4200 /wd26451 /wd26495 /D_CRT_SECURE_NO_WARNINGS /utf-8 2024-01-05T17:55:49.0688919Z 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed: None 2024-01-05T17:55:49.0690235Z error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed: None 2024-01-05T17:55:49.0691031Z [end of output] ```

When patched to dump the stack trace:

2024-01-05T18:46:26.5479903Z   Traceback (most recent call last):
2024-01-05T18:46:26.5485629Z     File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b6b5ns6s\overlay\Lib\site-packages\setuptools\_distutils\spawn.py", line 57, in spawn
2024-01-05T18:46:26.5486556Z       proc = subprocess.Popen(cmd, env=env)
2024-01-05T18:46:26.5486909Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-05T18:46:26.5487748Z     File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.0\tools\Lib\subprocess.py", line 1026, in __init__
2024-01-05T18:46:26.5488652Z       self._execute_child(args, executable, preexec_fn, close_fds,
2024-01-05T18:46:26.5489592Z     File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.0\tools\Lib\subprocess.py", line 1538, in _execute_child
2024-01-05T18:46:26.5490616Z       hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2024-01-05T18:46:26.5491062Z                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-05T18:46:26.5491559Z   FileNotFoundError: [WinError 206] The filename or extension is too long
2024-01-05T18:46:26.5491996Z   
2024-01-05T18:46:26.5492336Z   The above exception was the direct cause of the following exception:
2024-01-05T18:46:26.5492748Z   
2024-01-05T18:46:26.5492952Z   Traceback (most recent call last):
2024-01-05T18:46:26.5493379Z     File "<string>", line 56, in spawn_with_compiler_launcher
2024-01-05T18:46:26.5494346Z     File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b6b5ns6s\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 517, in spawn
2024-01-05T18:46:26.5495223Z       return super().spawn(cmd, env=env)
2024-01-05T18:46:26.5495538Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-05T18:46:26.5496382Z     File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b6b5ns6s\overlay\Lib\site-packages\setuptools\_distutils\ccompiler.py", line 1041, in spawn
2024-01-05T18:46:26.5497250Z       spawn(cmd, dry_run=self.dry_run, **kwargs)
2024-01-05T18:46:26.5498111Z     File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b6b5ns6s\overlay\Lib\site-packages\setuptools\_distutils\spawn.py", line 63, in spawn
2024-01-05T18:46:26.5498995Z       raise DistutilsExecError(
2024-01-05T18:46:26.5499949Z   distutils.errors.DistutilsExecError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed: None
Mause commented 10 months ago

Doing a diff on the commands generated by CPython's distutils and the one included with setuptools, it looks like the difference might be caused by the name of the build folder - build\temp.win-amd64-3.10\Release vs build\temp.win-amd64-cpython-312\Release. The inclusion of cpython in every single path definitely explains the explosion in size anyway. And turns out we pretty close to the limit even with the CPython copy of distutils, at around 30'000 characters

I guess the question here then is, do we need to migrate away from setuptools? Or is there some way we can split up the link calls?

abravalheri commented 10 months ago

Hi @Mause thank you very much for reporting this issue.

Is this something that can be solved by changing the LongPathsEnabled option in the Windows registra? For example:

# powershell
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

If not, I belive the best would be adding the workaround with the shortening function directly to the pypa/distutils repository (but probably the pypa/distutils maintainers will be the best people to advise on that).

Mause commented 10 months ago

Hi @Mause thank you very much for reporting this issue.

Is this something that can be solved by changing the LongPathsEnabled option in the Windows registra? For example:

# powershell
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

This is already enabled - without it, the limit is 320 characters I believe

If not, I belive the best would be adding the workaround with the shortening function directly to the pypa/distutils repository (but probably the pypa/distutils maintainers will be the best people to advise on that).

I'm happy to do this, but arguably this just delays the issue, rather than solving it - is there a way to pass link.exe arguments with a file perhaps?

abravalheri commented 10 months ago

is there a way to pass link.exe arguments with a file perhaps?

The microsoft docs do mention something like "link command files", so in theory yes that would be possible (but I don't know if that would be a solution or the behaviour is just to use the files as if they were directly typed in the terminal... that probably would have to be tested).

Other possibilities include cd-ing into a parent directory and use relative paths.

I believe that the folks in the pypa/distutils repository would be the best to advise on this.

/cc @zooba (for advice) /cc @jaraco (for advice, is this something to be handled in setuptools or distutils?)

@Mause, one quick question just to make sure we are looking into the correct bug: Is it expected for the repository in question that the file structure goes deep (more than 32766 chars, considering of course that setuptools will add the <repository root...>\build\temp.win-amd64-cpython-312\Release prefix), or is there any other bug causing the paths to be nested multiple times?

zooba commented 10 months ago

The compiler and linker don't support long paths, unfortunately.

If the problem is that the overall command line is too long, then a command file is a good option (also helps avoid quoting issues, though as we've mostly got those sorted it probably isn't a big deal anymore). But if the problem is that an individual path exceeds 260 then I don't think there's a way around it yet.

I'll prod the team about this again. Most likely they'll suggest moving the build directory to a shorter path (again)...

abravalheri commented 10 months ago

Thank you very much @zooba for the advice.

@Mause, this would mean that the "longest" term option we have available is to use the shortening path workaround as you described.

Regarding the path added by setuptools: <project root...>\build\temp.win-amd64-cpython-312 It part can be probably tweaked by setting build_base and build_temp in the [build] section of the setup.cfg file.

Is it expected for the repository in question that the file structure goes deep (more than 32766 chars, considering of course that setuptools will add ... prefix), or is there any other bug causing the paths to be nested multiple times?

But it might be worth checking if there is no other bug causing unnecessary nesting first.

Mause commented 10 months ago

The compiler and linker don't support long paths, unfortunately.

In my particular case that doesn't matter too much, as the longest path in our codebase is only 113 characters: build\temp.win-amd64-cpython-312\Release\duckdb_build\third_party\thrift\thrift\transport\TTransportException.obj

But it might be worth checking if there is no other bug causing unnecessary nesting first.

I've only eyeballed the paths in question (see my original post for the very long link.exe invocation in question), but I can't see any extra nesting beyond what was there prior in older versions of distutils.

It sounds to me like the link command files are a pretty viable option, and can be combined with the short path names to work around the fact that the compiler and linker don't support long paths as yet (though again, this second part doesn't affect us too much, as individually our paths aren't all that long).

abravalheri commented 10 months ago

It sounds to me like the link command files are a pretty viable option, and can be combined with the short path names to work around the fact that the compiler and linker don't support long paths as yet (though again, this second part doesn't affect us too much, as individually our paths aren't all that long).

I think the best would be discussing the problem in pypa/distutils and see if they would accept such a PR there. I say this because currently setuptools patches the functions for finding microsoft's compiler and toolkit, but not the functions for linking or compiling in the MSVCCompiler class.

as the longest path in our codebase is only 113 characters

This is interesting... so the file paths are actually not getting beyond the limit of 32766 that you get when you change the registry option...

Do you think it is because the command line string is too big? But then I think distutils calls the executables using Popen without passing through a shell... I am not sure if in that case the line length impacts that much... and the error message should be different, shouldn't it?

Mause commented 10 months ago

It sounds to me like the link command files are a pretty viable option, and can be combined with the short path names to work around the fact that the compiler and linker don't support long paths as yet (though again, this second part doesn't affect us too much, as individually our paths aren't all that long).

I think the best would be discussing the problem in pypa/distutils and see if they would accept such a PR there. I say this because currently setuptools patches the functions for finding microsoft's compiler and toolkit, but not the functions for linking or compiling in the MSVCCompiler class.

Can do

as the longest path in our codebase is only 113 characters

This is interesting... so the file paths are actually not getting beyond the limit of 32766 that you get when you change the registry option...

Do you think it is because the command line string is too big? But then I think distutils calls the executables using Popen without passing through a shell... I am not sure if in that case the line length impacts that much... and the error message should be different, shouldn't it?

Yes, that's what this issue is.

Under the hood on windows, Popen calls CreateProcessW.

From the Windows docs:

lpCommandLine

The command line to be executed.

The maximum length of this string is 32,767 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa#parameters