Open jacobwilliams opened 11 months ago
I faced the exact same problem when trying to build packages containing many subfolders. Looks like the issue comes from https://github.com/conda/conda-build/blob/46dfcf8dac6b456de05144625af796666fba21de/conda_build/build.py#L355-L357
In my case, patching with a fixed chunk size of 15000 worked, but it's maybe not optimal to lower the limit each time a user reaches the limit.
Something like:
file_lists = list(
chunks(prefix_files, (float(os.environ.get("CONDA_BLD_REGEX_CHUNK_SIZE", 15000)) if utils.on_win else 131071) - args_len)
)
would be preferable, but I don't know much about the best way to pass a config value to conda-build
and there is probably way for improvement here.
I'll open a PR to support this idea!
I'm curious to know the reason for these magic numbers (32760 & 131071, and now 15000).
Edit: I guess 32760 is supposed to be the max length of a command line argument in Windows (according to this).
BTW, this fix (just manually changing 32760 to 15000 works for me! Thanks!
I think this chunks
routine is fundamentally broken, since it allows lines that are > than these bounds. The fix should be to rewrite it so that it doesn't do that.
Example:
prefix_files = ['1', '2', '333', '4', '44', '55555', '1', '1', '1', '1', '1', '1', '1']
print('')
file_lists = list( chunks(prefix_files, 10) )
print(file_lists)
for f in file_lists:
print(len(''.join(f)))
produces:
[['1', '2', '333'], ['333', '4', '44', '55555'], ['55555', '1', '1', '1'], ['1', '1', '1', '1'], ['1', '1']]
5
11
8
4
2
Notice we already have one that is > 10 right there.
Possible rewrite of this routine (I don't know what yield
does, so I wrote it a simpler way. note it returns a list):
def chunks(line: list[str], n: int):
_PADDING = 3 # + 3 in case a shell is used: 1 space and 2 quotes.
result = [] # the list that will be returned
element = [] # an element (list) in the returned list
size = 0 # size accumulator for each element
for f in line:
f_size = len(f) + _PADDING # padded size of current f
tmp_size = size + f_size # size if we added current f to element
if tmp_size > n: # the chunk would be too big if we add this one
if element: # done with this chunk, add what we have so far
result.append(element)
size = f_size # first one in the next chunk
element = [f]
else: # add another one to this chunk
size = tmp_size
element.append(f)
if element: # last one if necessary
result.append(element)
return result
produces:
[['1', '2'], ['333', '4'], ['44'], ['55555'], ['1', '1'], ['1', '1'], ['1', '1'], ['1']]
2
4
2
5
2
2
2
1
So, none of them are > 10.
Note that compile_missing_pyc()
seems to be assuming the max for Windows is 8190.
# We avoid command lines longer than 8190
if on_win:
limit = 8190
else:
limit = 32760
Checklist
What happened?
When running
conda build
for my package on Windows, I sometimes get this:Sometimes rebooting the computer make it work, but not always. On the theory that i have file names that are too long, I shortened some of them, and that made it work for a while, but now it doesn't work anymore. It seems somewhat random. My package is huge and has many files. I've had this problem for the last several versions of conda and conda-build that I have tried.
Any ideas or suggestions? Seems like a Windows-specific issue (the same package builds fine on Linux and Mac).
Conda Info
Conda Config
Conda list
```shell # Name Version Build Channel astunparse 1.6.3 pyhd8ed1ab_0 conda-forge atomicwrites 1.4.1 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge aws-c-auth 0.7.0 h6f3c987_2 conda-forge aws-c-cal 0.6.0 h6ba3258_0 conda-forge aws-c-common 0.8.23 hcfcfb64_0 conda-forge aws-c-compression 0.2.17 h420beca_1 conda-forge aws-c-event-stream 0.3.1 had47b81_1 conda-forge aws-c-http 0.7.11 h72ba615_0 conda-forge aws-c-io 0.13.28 ha35c040_0 conda-forge aws-c-mqtt 0.8.14 h4941efa_2 conda-forge aws-c-s3 0.3.13 he04eaa7_2 conda-forge aws-c-sdkutils 0.1.11 h420beca_1 conda-forge aws-checksums 0.1.16 h420beca_1 conda-forge aws-crt-cpp 0.20.3 h247a981_4 conda-forge aws-sdk-cpp 1.10.57 h1a0519f_17 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge blis 0.9.0 hcfcfb64_1 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge boltons 23.0.0 pyhd8ed1ab_0 conda-forge boost-cpp 1.78.0 h9f4b32c_4 conda-forge brotli 1.0.9 hcfcfb64_9 conda-forge brotli-bin 1.0.9 hcfcfb64_9 conda-forge brotli-python 1.0.9 py310h00ffb61_9 conda-forge build 0.7.0 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 hcfcfb64_5 conda-forge c-ares 1.24.0 hcfcfb64_0 conda-forge ca-certificates 2023.11.17 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.16.0 hd694305_1014 conda-forge certifi 2023.11.17 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py310h8d17308_0 conda-forge chardet 5.2.0 py310h5588dad_1 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge click 8.1.3 win_pyhd8ed1ab_2 conda-forge cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge collada-dom 2.5.0 hcb3fdb8_5 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge conda 23.7.4 py310h5588dad_0 conda-forge conda-build 3.23.3 py310h5588dad_1 conda-forge conda-package-handling 2.2.0 pyh38be061_0 conda-forge conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge conda-verify 3.1.1 py310h5588dad_1006 conda-forge coverage 7.2.1 py310h8d17308_0 conda-forge cryptography 41.0.7 py310hb1f9477_1 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cyclonedx-bom 3.11.0 pyhd8ed1ab_0 conda-forge cyclonedx-python-lib 3.1.5 pyhd8ed1ab_0 conda-forge cytoolz 0.12.2 py310h8d17308_1 conda-forge dask 2023.3.2 pyhd8ed1ab_0 conda-forge dask-core 2023.3.2 pyhd8ed1ab_0 conda-forge dask-jobqueue 0.8.1 pyhd8ed1ab_0 conda-forge distributed 2023.3.2.1 pyhd8ed1ab_0 conda-forge dpcpp-cpp-rt 2023.0.0 h63175ca_25922 conda-forge expat 2.5.0 h63175ca_1 conda-forge f90nml 1.3.1 pyhd8ed1ab_0 conda-forge fastnml 2.0.0 pyhd8ed1ab_0 conda-forge ffmpeg 4.3.1 ha925a31_0 conda-forge filelock 3.13.1 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_1 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.47.0 py310h8d17308_0 conda-forge fortran_rt 2023.0.0 intel_25922 intel freetype 2.12.1 hdaf720e_2 conda-forge fribidi 1.0.10 h8d14728_0 conda-forge fsspec 2023.12.2 pyhca7485f_0 conda-forge future 0.18.3 pyhd8ed1ab_0 conda-forge getopt-win32 0.1 hcfcfb64_1 conda-forge gettext 0.21.1 h5728263_0 conda-forge glib 2.78.3 h12be248_0 conda-forge glib-tools 2.78.3 h12be248_0 conda-forge glob2 0.7 py_0 conda-forge graphite2 1.3.13 1000 conda-forge graphviz 2.50.0 had6c3a3_3 conda-forge gst-plugins-base 1.22.7 h001b923_1 conda-forge gstreamer 1.22.7 hb4038d2_1 conda-forge gts 0.7.6 h6b5321d_4 conda-forge h5py 3.7.0 nompi_py310hae7a582_102 conda-forge harfbuzz 6.0.0 he256f1b_0 conda-forge hdf5 1.12.2 nompi_h57737ce_101 conda-forge icc_rt 2023.0.0 intel_25922 intel icu 70.1 h0e60522_0 conda-forge idna 3.6 pyhd8ed1ab_0 conda-forge importlib-metadata 7.0.0 pyha770c72_0 conda-forge importlib_metadata 7.0.0 hd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge intel-cmplr-lib-rt 2023.0.0 h12be248_25922 conda-forge intel-cmplr-lic-rt 2023.0.0 h57928b3_25922 conda-forge intel-fortran-rt 2023.0.0 hcfcfb64_25922 conda-forge intel-opencl-rt 2023.0.0 h927a3ec_25922 conda-forge intel-openmp 2023.0.0 intel_25922 intel jedi 0.18.1 pyhd8ed1ab_2 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jpeg 9e h8ffe710_2 conda-forge jsonpatch 1.33 pyhd8ed1ab_0 conda-forge jsonpointer 2.4 py310h5588dad_3 conda-forge khronos-opencl-icd-loader 2023.04.17 h64bf75a_0 conda-forge kiwisolver 1.4.5 py310h232114e_1 conda-forge krb5 1.20.1 heb0366b_0 conda-forge lcms2 2.12 h2a16943_0 conda-forge lerc 3.0 h0e60522_0 conda-forge libabseil 20230125.3 cxx17_h63175ca_0 conda-forge libaec 1.1.2 h63175ca_1 conda-forge libarchive 3.6.2 h27c7867_0 conda-forge libarrow 12.0.1 h0578746_5_cpu conda-forge libblas 3.9.0 20_win64_blis conda-forge libbrotlicommon 1.0.9 hcfcfb64_9 conda-forge libbrotlidec 1.0.9 hcfcfb64_9 conda-forge libbrotlienc 1.0.9 hcfcfb64_9 conda-forge libcblas 3.9.0 20_win64_blis conda-forge libclang 15.0.7 default_h77d9078_3 conda-forge libclang13 15.0.7 default_h77d9078_3 conda-forge libcrc32c 1.1.2 h0e60522_0 conda-forge libcurl 8.1.2 h68f0423_0 conda-forge libdeflate 1.10 h8ffe710_0 conda-forge libevent 2.1.12 h3671451_1 conda-forge libexpat 2.5.0 h63175ca_1 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libgd 2.3.3 h891f43f_3 conda-forge libglib 2.78.3 h16e383f_0 conda-forge libgoogle-cloud 2.12.0 hbc1b25b_1 conda-forge libgrpc 1.56.2 hea2d5f7_1 conda-forge libhwloc 2.9.1 h51c2c0f_0 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge libjpeg-turbo 2.1.4 hcfcfb64_0 conda-forge liblapack 3.9.0 5_hd5c7e75_netlib conda-forge liblief 0.12.3 h63175ca_0 conda-forge libogg 1.3.4 h8ffe710_1 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libprotobuf 4.23.3 h1975477_1 conda-forge libsqlite 3.44.2 hcfcfb64_0 conda-forge libssh2 1.11.0 h7dfc565_0 conda-forge libthrift 0.18.1 h06f6336_2 conda-forge libtiff 4.3.0 hc4061b1_4 conda-forge libutf8proc 2.8.0 h82a8f57_0 conda-forge libvorbis 1.3.7 h0e60522_0 conda-forge libwebp 1.3.2 hcfcfb64_1 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.15 hcd874cb_0 conda-forge libxml2 2.10.4 hc3477c8_0 conda-forge libxslt 1.1.37 h0192164_0 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge llvm-openmp 17.0.6 h91493d7_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4 4.3.2 py310hbbb2075_1 conda-forge lz4-c 1.9.4 hcfcfb64_0 conda-forge lzo 2.10 he774522_1000 conda-forge m2-msys2-runtime 2.5.0.17080.65c939c 3 conda-forge m2-patch 2.7.5 2 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.1 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h8d17308_1 conda-forge matplotlib-base 3.5.1 py310h79a7439_0 conda-forge menuinst 1.4.20 py310h00ffb61_0 conda-forge mkl 2023.0.0 intel_25930 intel msgpack-python 1.0.5 py310h232114e_1 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge networkx 3.0 pyhd8ed1ab_0 conda-forge nodejs 18.12.1 h57928b3_0 conda-forge numpy 1.22.1 py310hcae7c84_0 conda-forge olefile 0.47 pyhd8ed1ab_0 conda-forge openjpeg 2.5.0 hb211442_0 conda-forge openscenegraph 3.6.5 h145966f_13 conda-forge openssl 3.2.0 hcfcfb64_1 conda-forge orc 1.9.0 hf2b8f0d_1 conda-forge packageurl-python 0.13.1 pyhd8ed1ab_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 1.4.0 py310hf5e1058_0 conda-forge pango 1.50.14 hdffb7b3_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge pcre 8.45 h0e60522_0 conda-forge pcre2 10.42 h17e33f8_0 conda-forge pdoc 13.0.0 pyhd8ed1ab_0 conda-forge pep517 0.13.0 pyhd8ed1ab_0 conda-forge pillow 8.4.0 py310h22f3323_0 conda-forge pip 23.3.2 pyhd8ed1ab_0 conda-forge pip-requirements-parser 32.0.1 pyhd8ed1ab_0 conda-forge pixman 0.42.2 h63175ca_0 conda-forge pkginfo 1.9.6 pyhd8ed1ab_0 conda-forge platformdirs 4.1.0 pyhd8ed1ab_0 conda-forge pluggy 1.3.0 pyhd8ed1ab_0 conda-forge pooch 1.8.0 pyhd8ed1ab_0 conda-forge psutil 5.9.0 py310he2412df_1 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge py 1.11.0 pyh6c4a22f_0 conda-forge py-lief 0.12.3 py310h00ffb61_0 conda-forge pyarrow 12.0.1 py310hd1a9178_5_cpu conda-forge pycosat 0.6.6 py310h8d17308_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyopenssl 23.3.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyqtconsole 1.2.2 pyhd8ed1ab_0 conda-forge pyqtgraph 0.13.1 pyhd8ed1ab_1 conda-forge pyshortcuts 1.8.1 py310h5588dad_0 conda-forge pyside2 5.15.7 py310h376e08a_0 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge pytest 6.2.5 py310h5588dad_3 conda-forge pytest-qt 4.0.2 pyhd8ed1ab_0 conda-forge python 3.10.13 h4de0772_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-libarchive-c 5.0 py310h5588dad_2 conda-forge python_abi 3.10 4_cp310 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pywin32 303 py310h8d17308_1 conda-forge pyyaml 6.0.1 py310h8d17308_1 conda-forge qt-main 5.15.8 h720456b_6 conda-forge qtpy 2.0.0 pyhd8ed1ab_0 conda-forge re2 2023.03.02 hd4eee63_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge ripgrep 14.0.3 h7f3b576_1 conda-forge ruamel.yaml 0.17.40 py310h8d17308_0 conda-forge ruamel.yaml.clib 0.2.7 py310h8d17308_2 conda-forge scipy 1.10.1 py310h578b7cb_3 conda-forge setuptools 65.6.3 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 hfb803bf_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tbb 2021.9.0 h91493d7_0 conda-forge tblib 3.0.0 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5226925_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h8d17308_1 conda-forge tqdm 4.66.1 pyhd8ed1ab_0 conda-forge typing_extensions 4.9.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 15.1.0 py310h8d17308_0 conda-forge urllib3 2.1.0 pyhd8ed1ab_0 conda-forge vc 14.3 hcf57466_18 conda-forge vc14_runtime 14.38.33130 h82b7239_18 conda-forge vs2015_runtime 14.38.33130 hcb4865c_18 conda-forge wheel 0.42.0 pyhd8ed1ab_0 conda-forge win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge xorg-kbproto 1.0.7 hcd874cb_1002 conda-forge xorg-libice 1.1.1 hcd874cb_0 conda-forge xorg-libsm 1.2.4 hcd874cb_0 conda-forge xorg-libx11 1.8.7 hefa74cf_0 conda-forge xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xorg-libxext 1.3.4 hcd874cb_2 conda-forge xorg-libxpm 3.5.17 hcd874cb_0 conda-forge xorg-libxt 1.3.0 hcd874cb_1 conda-forge xorg-xextproto 7.3.0 hcd874cb_1003 conda-forge xorg-xproto 7.0.31 hcd874cb_1007 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hcfcfb64_5 conda-forge zstandard 0.22.0 py310h0009e47_0 conda-forge zstd 1.5.5 h12be248_0 conda-forge ```conda list
Additional Context
No response