search

conda-forge / conda-smithy

The tool for managing conda-forge feedstocks.
https://conda-forge.org/
BSD 3-Clause "New" or "Revised" License
147 stars 177 forks source link

v3.32.0 requires jsonschema #1866

Closed jdblischak closed 5 months ago

jdblischak commented 5 months ago

Solution to issue cannot be found in the documentation.

Issue

The recently released v3.32.0 included PR https://github.com/conda-forge/conda-smithy/pull/1756, which added a requirement on jsonschema. jsonschema was added to environment.yml, but not pyproject.toml

https://github.com/conda-forge/conda-smithy/blob/85ddb5deabda9ab58ae6ec898e2a1e33393117b7/environment.yml#L36

jsonschema was also not added as a requirement in the automerged feedstock PR https://github.com/conda-forge/conda-smithy-feedstock/pull/281

This has caused my CI pipelines that run conda smithy rerender to fail with the following error:

Traceback (most recent call last):
  File "/home/runner/micromamba/envs/env/bin/conda-smithy", line 6, in <module>
    from conda_smithy.cli import main
  File "/home/runner/micromamba/envs/env/lib/python3.11/site-packages/conda_smithy/cli.py", line 20, in <module>
    from . import configure_feedstock
  File "/home/runner/micromamba/envs/env/lib/python3.11/site-packages/conda_smithy/configure_feedstock.py", line [12](https://github.com/TileDB-Inc/conda-forge-nightly-controller/actions/runs/8289914269/job/22688574049#step:14:13), in <module>
    import jsonschema
ModuleNotFoundError: No module named 'jsonschema'

Shouldn't jsonschema be added at minimum to the conda recipe requirements?

xref: https://github.com/TileDB-Inc/conda-forge-nightly-controller/issues/64, https://github.com/TileDB-Inc/conda-forge-nightly-controller/issues/65, https://github.com/TileDB-Inc/tiledbsoma-feedstock/issues/100

Installed packages

/home/runner/micromamba-bin/micromamba list -r /home/runner/micromamba -n env
  List of packages in environment: "/home/runner/micromamba/envs/env"

    Name                     Version              Build               Channel    
  ─────────────────────────────────────────────────────────────────────────────────
    _libgcc_mutex            0.1                  conda_forge         conda-forge
    _openmp_mutex            4.5                  2_gnu               conda-forge
    beautifulsoup4           4.12.2               pyha770c72_0        conda-forge
    blinker                  1.6.3                pyhd8ed1ab_0        conda-forge
    boltons                  23.0.0               pyhd8ed1ab_0        conda-forge
    boolean.py               4.0                  pyhd8ed1ab_0        conda-forge
    brotli-python            1.1.0                py311hb755f60_1     conda-forge
    bzip2                    1.0.8                h7f98852_4          conda-forge
    c-ares                   1.20.1               hd590300_0          conda-forge
    ca-certificates          2023.7.22            hbcca054_0          conda-forge
    certifi                  2023.7.22            pyhd8ed1ab_0        conda-forge
    cffi                     1.16.0               py311hb3a22ac_0     conda-forge
    chardet                  5.2.0                py311h38be061_1     conda-forge
    charset-normalizer       3.3.0                pyhd8ed1ab_0        conda-forge
    click                    8.1.7                unix_pyh707e725_0   conda-forge
    colorama                 0.4.6                pyhd8ed1ab_0        conda-forge
    conda                    23.9.0               py311h38be061_0     conda-forge
    conda-build              3.27.0               py311h38be061_0     conda-forge
    conda-forge-pinning      2023.10.11.19.37.54  hd8ed1ab_0          conda-forge
    conda-index              0.2.3                pyhd8ed1ab_0        conda-forge
    conda-package-handling   2.2.0                pyh38be061_0        conda-forge
    conda-package-streaming  0.9.0                pyhd8ed1ab_0        conda-forge
    conda-smithy             3.27.1               pyhd8ed1ab_0        conda-forge
    cryptography             41.0.4               py311h63ff55d_0     conda-forge
    curl                     8.4.0                hca28451_0          conda-forge
    deprecated               1.2.14               pyh1a96a4e_0        conda-forge
    filelock                 3.12.4               pyhd8ed1ab_0        conda-forge
    gettext                  0.21.1               h27087fc_0          conda-forge
    git                      2.42.0               pl5321h86e50cf_0    conda-forge
    gitdb                    4.0.10               pyhd8ed1ab_0        conda-forge
    gitpython                3.1.37               pyhd8ed1ab_0        conda-forge
    gmp                      6.2.1                h58526e2_0          conda-forge
    icu                      73.2                 h59595ed_0          conda-forge
    idna                     3.4                  pyhd8ed1ab_0        conda-forge
    isodate                  0.6.1                pyhd8ed1ab_0        conda-forge
    jinja2                   3.1.2                pyhd8ed1ab_1        conda-forge
    jsonpatch                1.33                 pyhd8ed1ab_0        conda-forge
    jsonpointer              2.4                  py311h38be061_3     conda-forge
    keyutils                 1.6.1                h166bdaf_0          conda-forge
    krb5                     1.21.2               h659d440_0          conda-forge
    ld_impl_linux-64         2.40                 h41732ed_0          conda-forge
    libarchive               3.7.2                h039dbb9_0          conda-forge
    libcurl                  8.4.0                hca28451_0          conda-forge
    libedit                  3.1.20191231         he28a2e2_2          conda-forge
    libev                    4.33                 h516909a_1          conda-forge
    libexpat                 2.5.0                hcb278e6_1          conda-forge
    libffi                   3.4.2                h7f98852_5          conda-forge
    libgcc-ng                13.2.0               h807b86a_2          conda-forge
    libgomp                  13.2.0               h807b86a_2          conda-forge
    libiconv                 1.17                 h166bdaf_0          conda-forge
    liblief                  0.12.3               h27087fc_0          conda-forge
    libnghttp2               1.52.0               h61bc06f_0          conda-forge
    libnsl                   2.0.0                hd590300_1          conda-forge
    libsodium                1.0.18               h36c2ea0_1          conda-forge
    libsqlite                3.43.2               h2797004_0          conda-forge
    libssh2                  1.11.0               h0841786_0          conda-forge
    libstdcxx-ng             13.2.0               h7e041cc_2          conda-forge
    libuuid                  2.38.1               h0b41bf4_0          conda-forge
    libxml2                  2.11.5               h232c23b_1          conda-forge
    libzlib                  1.2.13               hd590300_5          conda-forge
    license-expression       30.1.1               pyhd8ed1ab_0        conda-forge
    lz4-c                    1.9.4                hcb278e6_0          conda-forge
    lzo                      2.10                 h516909a_1000       conda-forge
    markupsafe               2.1.3                py311h459d7ec_1     conda-forge
    more-itertools           10.1.0               pyhd8ed1ab_0        conda-forge
    msrest                   0.6.21               pyh44b312d_0        conda-forge
    ncurses                  6.4                  hcb278e6_0          conda-forge
    oauthlib                 3.2.2                pyhd8ed1ab_0        conda-forge
    openssl                  3.1.3                hd590300_0          conda-forge
    packaging                23.2                 pyhd8ed1ab_0        conda-forge
    patch                    2.7.6                h7f98852_1002       conda-forge
    patchelf                 0.17.2               h58526e2_0          conda-forge
    pcre2                    10.40                hc3806b6_0          conda-forge
    perl                     5.32.1               4_hd590300_perl5    conda-forge
    pip                      23.2.1               pyhd8ed1ab_0        conda-forge
    pkginfo                  1.9.6                pyhd8ed1ab_0        conda-forge
    pluggy                   1.3.0                pyhd8ed1ab_0        conda-forge
    psutil                   5.9.5                py311h459d7ec_1     conda-forge
    py-lief                  0.12.3               py311ha362b79_0     conda-forge
    pycosat                  0.6.6                py311h459d7ec_0     conda-forge
    pycparser                2.21                 pyhd8ed1ab_0        conda-forge
    pycryptodome             3.19.0               py311h6601440_1     conda-forge
    pygithub                 1.59.1               pyhd8ed1ab_0        conda-forge
    pyjwt                    2.8.0                pyhd8ed1ab_0        conda-forge
    pynacl                   1.5.0                py311h459d7ec_3     conda-forge
    pyopenssl                23.2.0               pyhd8ed1ab_1        conda-forge
    pysocks                  1.7.1                pyha2e5f31_6        conda-forge
    python                   3.11.6               hab00c5b_0_cpython  conda-forge
    python-libarchive-c      5.0                  py311h38be061_2     conda-forge
    python_abi               3.11                 4_cp311             conda-forge
    pytz                     2023.3.post1         pyhd8ed1ab_0        conda-forge
    pyyaml                   6.0.1                py311h459d7ec_1     conda-forge
    readline                 8.2                  h8228510_1          conda-forge
    requests                 2.31.0               pyhd8ed1ab_0        conda-forge
    requests-oauthlib        1.3.1                pyhd8ed1ab_0        conda-forge
    ripgrep                  13.0.0               h2f28480_2          conda-forge
    ruamel.yaml              0.17.35              py311h459d7ec_0     conda-forge
    ruamel.yaml.clib         0.2.7                py311h459d7ec_2     conda-forge
    scrypt                   0.8.20               py311h6965a6d_1     conda-forge
    setuptools               68.2.2               pyhd8ed1ab_0        conda-forge
    six                      1.16.0               pyh6c4a22f_0        conda-forge
    smmap                    3.0.5                pyh44b312d_0        conda-forge
    soupsieve                2.5                  pyhd8ed1ab_1        conda-forge
    tk                       8.6.13               h2797004_0          conda-forge
    tomli                    2.0.1                pyhd8ed1ab_0        conda-forge
    toolz                    0.12.0               pyhd8ed1ab_0        conda-forge
    tqdm                     4.66.1               pyhd8ed1ab_0        conda-forge
    truststore               0.8.0                pyhd8ed1ab_0        conda-forge
    typing_extensions        4.8.0                pyha770c72_0        conda-forge
    tzdata                   2023c                h71feb2d_0          conda-forge
    urllib3                  2.0.6                pyhd8ed1ab_0        conda-forge
    vsts-python-api          0.1.25               pyhd8ed1ab_1        conda-forge
    wheel                    0.41.2               pyhd8ed1ab_0        conda-forge
    wrapt                    1.15.0               py311h459d7ec_1     conda-forge
    xz                       5.2.6                h166bdaf_0          conda-forge
    yaml                     0.2.5                h7f98852_2          conda-forge
    zstandard                0.21.0               py311haa97af0_1     conda-forge
    zstd                     1.5.5                hfc55251_0          conda-forge

Environment info

/home/runner/micromamba-bin/micromamba info -r /home/runner/micromamba -n env

         libmamba version : 1.5.7
       micromamba version : 1.5.7
             curl version : libcurl/8.5.0 OpenSSL/3.2.1 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0
       libarchive version : libarchive 3.7.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.5
         envs directories : /home/runner/micromamba/envs
            package cache : /home/runner/micromamba/pkgs
                            /home/runner/.mamba/pkgs
              environment : env
             env location : /home/runner/micromamba/envs/env
        user config files : /home/runner/.mambarc
   populated config files : /home/runner/work/_temp/setup-micromamba/.condarc
         virtual packages : __unix=0=0
                            __linux=6.5.0=0
                            __glibc=2.35=0
                            __archspec=1=x86_64-v3
                 channels : https://conda.anaconda.org/conda-forge/linux-64
                            https://conda.anaconda.org/conda-forge/noarch
         base environment : /home/runner/micromamba
                 platform : linux-64
jakirkham commented 5 months ago

cc @viniciusdc

jdblischak commented 5 months ago

I added jsonschema to the run requirements of the conda-forge recipe in https://github.com/conda-forge/conda-smithy-feedstock/pull/282

@beckermr volunteered to send the repodata-patch to patch the metadata of the binary from yesterday

What about pyproject.toml? That seems like a natural place to define run requirements. Is there a reason the run requirements are only specified in environment.yml?

beckermr commented 5 months ago

So we're more "conda first" here typically. The pyproject.toml usually lists dependencies from pypi, not say conda-forge, at least in the normal spots. Items on pypi don't always have the same name as they do in conda-forge so this can create issues and confusion. Some items are missing too. Some are not there on purpose: https://pypi.org/project/conda/

That all being said, projects like pixi are blurring the boundaries more which is both cool to see and certainly one of many possible paths forward.

jdblischak commented 5 months ago

The pyproject.toml usually lists dependencies from pypi, not say conda-forge, at least in the normal spots. Items on pypi don't always have the same name as they do in conda-forge so this can create issues and confusion.

@beckermr Thanks for the explanation! That makes sense. I see now that the conda-forge recipe uses GitHub releases as the source, and not PyPI, so I agree that maintaining a pyproject.toml isn't necessary.