search

conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
382 stars 423 forks source link

New debug logs present in 3.28.0 #5095

Closed JeanChristopheMorinPerso closed 9 months ago

JeanChristopheMorinPerso commented 11 months ago

Checklist

What happened?

conda-build 3.28.0 fixed a problem where there was debug logs like DEBUG:conda_index.index were printed to stdout.

Unfortunately, 3.28.0 introduces new debug logs.

(base) jcmorin-mac-anac:aggregate jcmorin$ conda-build jcmorin/transformers-feedstock --python 3.11
/Users/jcmorin/miniconda/bin/conda-build:11: DeprecationWarning: conda_build.cli.main_build.main is deprecated and will be removed in 24.1.0. Use `conda build` instead.
  sys.exit(main())
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Adding in variants from /Users/jcmorin/anaenv/aggregate/conda_build_config.yaml
Adding in variants from config.variant
Attempting to finalize metadata for transformers
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
BUILD START: ['transformers-4.31.0-py311hca03da5_0.tar.bz2']
DEBUG:chardet.charsetprober:SHIFT_JIS Japanese prober hit error at byte 2124
DEBUG:chardet.charsetprober:EUC-JP Japanese prober hit error at byte 1956
DEBUG:chardet.charsetprober:GB2312 Chinese prober hit error at byte 2127
DEBUG:chardet.charsetprober:EUC-KR Korean prober hit error at byte 1956
DEBUG:chardet.charsetprober:CP949 Korean prober hit error at byte 1956
DEBUG:chardet.charsetprober:Big5 Chinese prober hit error at byte 1957
DEBUG:chardet.charsetprober:EUC-TW Taiwan prober hit error at byte 1956
DEBUG:chardet.charsetprober:windows-1251 Russian confidence = 0.01
DEBUG:chardet.charsetprober:KOI8-R Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-5 Russian confidence = 0.0
DEBUG:chardet.charsetprober:MacCyrillic Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM866 Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM855 Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-7 Greek confidence = 0.0
DEBUG:chardet.charsetprober:windows-1253 Greek confidence = 0.0
DEBUG:chardet.charsetprober:ISO-8859-5 Bulgarian confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Bulgarian confidence = 0.01
DEBUG:chardet.charsetprober:TIS-620 Thai confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-9 Turkish confidence = 0.5409487892112278
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Russian confidence = 0.01
DEBUG:chardet.charsetprober:KOI8-R Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-5 Russian confidence = 0.0
DEBUG:chardet.charsetprober:MacCyrillic Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM866 Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM855 Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-7 Greek confidence = 0.0
DEBUG:chardet.charsetprober:windows-1253 Greek confidence = 0.0
DEBUG:chardet.charsetprober:ISO-8859-5 Bulgarian confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Bulgarian confidence = 0.01
DEBUG:chardet.charsetprober:TIS-620 Thai confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-9 Turkish confidence = 0.5409487892112278
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done

And for a reason that seems obscure to me, they don't appear on all feedstocks...

I'm currently getting these with https://github.com/AnacondaRecipes/transformers-feedstock/pull/9 but I'm not seeing them on other feedstocks.

Conda Info

active environment : base
    active env location : /Users/jcmorin/miniconda
            shell level : 1
       user config file : /Users/jcmorin/.condarc
 populated config files : /Users/jcmorin/.condarc
          conda version : 23.11.0
    conda-build version : 3.28.0
         python version : 3.11.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m2
                          __conda=23.11.0=0
                          __osx=13.6=0
                          __unix=0=0
       base environment : /Users/jcmorin/miniconda  (writable)
      conda av data dir : /Users/jcmorin/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/jcmorin/miniconda/pkgs
                          /Users/jcmorin/.conda/pkgs
       envs directories : /Users/jcmorin/miniconda/envs
                          /Users/jcmorin/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Darwin/22.6.0 OSX/13.6 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/jcmorin/.condarc <==
report_errors: False
conda-build: {}

Conda list

# packages in environment at /Users/jcmorin/miniconda:
#
# Name                    Version                   Build  Channel
archspec                  0.2.1              pyhd3eb1b0_0    defaults
attrs                     23.1.0          py311hca03da5_0    defaults
backports                 1.1                pyhd3eb1b0_0    defaults
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0    defaults
backports.tempfile        1.0                pyhd3eb1b0_1    defaults
backports.weakref         1.0.post1                  py_1    defaults
beautifulsoup4            4.12.2          py311hca03da5_0    defaults
boltons                   23.0.0          py311hca03da5_0    defaults
brotli-python             1.0.9           py311h313beb8_7    defaults
brotlipy                  0.7.0           py311h80987f9_1002    defaults
bzip2                     1.0.8                h620ffc9_4    defaults
c-ares                    1.19.1               h80987f9_0    defaults
ca-certificates           2023.08.22           hca03da5_0    defaults
cctools                   949.0.1             hc179dcd_25    defaults
cctools_osx-arm64         949.0.1             h332cad3_25    defaults
certifi                   2023.11.17      py311hca03da5_0    defaults
cffi                      1.16.0          py311h80987f9_0    defaults
chardet                   4.0.0           py311hca03da5_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.1.7           py311hca03da5_0    defaults
conda                     23.11.0         py311hca03da5_0    defaults
conda-build               3.28.0          py311hca03da5_0    defaults
conda-content-trust       0.2.0           py311hca03da5_0    defaults
conda-index               0.3.0           py311hca03da5_0    defaults
conda-libmamba-solver     23.11.1         py311hca03da5_0    defaults
conda-package-handling    2.2.0           py311hca03da5_0    defaults
conda-package-streaming   0.9.0           py311hca03da5_0    defaults
conda-verify              3.4.2                      py_1    defaults
cryptography              41.0.3          py311hd4332d6_0    defaults
darts                     0.25.0                   pypi_0    pypi
distro                    1.8.0           py311hca03da5_0    defaults
filelock                  3.13.1          py311hca03da5_0    defaults
fmt                       9.1.0                h48ca7d4_0    defaults
future                    0.18.3          py311hca03da5_0    defaults
glob2                     0.7                pyhd3eb1b0_0    defaults
icu                       73.1                 h313beb8_0    defaults
idna                      3.4             py311hca03da5_0    defaults
jinja2                    3.1.2           py311hca03da5_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
jsonschema                4.19.2          py311hca03da5_0    defaults
jsonschema-specifications 2023.7.1        py311hca03da5_0    defaults
krb5                      1.20.1               hf3e1bf2_1    defaults
ld64                      530                 hb29bf3f_25    defaults
ld64_osx-arm64            530                 h001ce53_25    defaults
ldid                      2.1.5                h20b2a84_3    defaults
libarchive                3.6.2                h62fee54_2    defaults
libcurl                   8.4.0                h3e2b118_0    defaults
libcxx                    14.0.6               h848a8c0_0    defaults
libedit                   3.1.20221030         h80987f9_0    defaults
libev                     4.33                 h1a28f6b_1    defaults
libffi                    3.4.4                hca03da5_0    defaults
libiconv                  1.16                 h1a28f6b_2    defaults
liblief                   0.12.3               h313beb8_0    defaults
libllvm14                 14.0.6               h7ec7a93_3    defaults
libmamba                  1.5.3                h15e39b3_0    defaults
libmambapy                1.5.3           py311h1c5506f_0    defaults
libnghttp2                1.57.0               h62f6fdd_0    defaults
libsolv                   0.7.24               h514c7bf_0    defaults
libssh2                   1.10.0               h02f6b3c_2    defaults
libxml2                   2.10.4               h0dcf63f_1    defaults
lz4-c                     1.9.4                h313beb8_0    defaults
markupsafe                2.1.1           py311h80987f9_0    defaults
menuinst                  2.0.0           py311hca03da5_0    defaults
more-itertools            10.1.0          py311hca03da5_0    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   3.0.12               h1a28f6b_0    defaults
packaging                 23.1            py311hca03da5_0    defaults
patch                     2.7.6             h1a28f6b_1001    defaults
pcre2                     10.42                hb066dcc_0    defaults
pip                       23.3.1          py311hca03da5_0    defaults
pkginfo                   1.9.6           py311hca03da5_0    defaults
platformdirs              3.10.0          py311hca03da5_0    defaults
pluggy                    1.0.0           py311hca03da5_1    defaults
psutil                    5.9.0           py311h80987f9_0    defaults
py-lief                   0.12.3          py311h313beb8_0    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py311h80987f9_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 23.2.0          py311hca03da5_0    defaults
pysocks                   1.7.1           py311hca03da5_0    defaults
python                    3.11.5               hb885b13_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
python.app                3               py311h80987f9_0    defaults
pytz                      2023.3.post1    py311hca03da5_0    defaults
pyyaml                    6.0.1           py311h80987f9_0    defaults
readline                  8.2                  h1a28f6b_0    defaults
referencing               0.30.2          py311hca03da5_0    defaults
reproc                    14.2.4               hc377ac9_1    defaults
reproc-cpp                14.2.4               hc377ac9_1    defaults
requests                  2.31.0          py311hca03da5_0    defaults
rpds-py                   0.10.6          py311hf0e4da2_0    defaults
ruamel.yaml               0.17.21         py311h80987f9_0    defaults
ruamel.yaml.clib          0.2.7           py311h80987f9_0    defaults
setuptools                68.0.0          py311hca03da5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
soupsieve                 2.5             py311hca03da5_0    defaults
sqlite                    3.41.2               h80987f9_0    defaults
tapi                      1100.0.11            h8754e6a_1    defaults
tk                        8.6.12               hb8d0fd4_0    defaults
tomli                     2.0.1           py311hca03da5_0    defaults
toolz                     0.12.0          py311hca03da5_0    defaults
tqdm                      4.65.0          py311hb6e6a13_0    defaults
truststore                0.8.0           py311hca03da5_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.18         py311hca03da5_0    defaults
wheel                     0.41.2          py311hca03da5_0    defaults
xz                        5.4.2                h80987f9_0    defaults
yaml                      0.2.5                h1a28f6b_0    defaults
yaml-cpp                  0.8.0                h313beb8_0    defaults
zlib                      1.2.13               h5a0b063_0    defaults
zstandard                 0.19.0          py311h80987f9_0    defaults
zstd                      1.5.5                hd90d995_0    defaults

Additional Context

No response

mbargull commented 11 months ago

And for a reason that seems obscure to me, they don't appear on all feedstocks...

I'm currently getting these with https://github.com/AnacondaRecipes/transformers-feedstock/pull/9 but I'm not seeing them on other feedstocks.

That recipe has some emoji characters:

    - 📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.
    - 🖼️ Images, for tasks like image classification, object detection, and segmentation.
    - 🗣️ Audio, for tasks like speech recognition and audio classification.

highly likely that those triggered that.

Unfortunately, 3.28.0 introduces new debug logs.

I don't believe this is any new behavior; maybe you can try some older versions to confirm this really is new?

The origin from these log messages is probably in bs4.dammit which tries a bunch of encoding before trying UTF-8, including those from chardet's prober. We probably want to replace all uses of UnicodeDammit with something that tries UTF-8 first (and then uses UnicodeDammit or something else if really necessary).