v3.32.0 cannot rerender feedstocks with custom Azure project configuration

jdblischak commented 5 months ago

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue

The TileDB-Inc GitHub org maintains independent feedstocks that we run in our own Azure org. In order to properly render the Azure status badges in the README, we add the fields build_id, project_name, and user_or_org underneath azure. This worked perfectly with conda-smithy <3.32.0.

Here is an example:

azure:
  build_id: 43
  user_or_org: TileDB-Inc
  project_name: CI

Below is a reprex:

git clone https://github.com/TileDB-Inc/tiledbsoma-feedstock.git
cd tiledbsoma-feedstock/
conda smithy --version
## 3.32.0
conda smithy rerender

  + Exception Group Traceback (most recent call last):
  |   File "/home/wsl/mambaforge/bin/conda-smithy", line 10, in <module>
  |     sys.exit(main())
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 733, in main
  |     args.subcommand_func(args)
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 584, in __call__
  |     self._call(args, tmpdir)
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 589, in _call
  |     configure_feedstock.main(
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 2537, in main
  |     config = _load_forge_config(forge_dir, exclusive_config_file, forge_yml)
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 2087, in _load_forge_config
  |     config = _read_forge_config(forge_dir, forge_yml=forge_yml)
  |   File "/home/wsl/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 2003, in _read_forge_config
  |     raise ExceptionGroup("lints", [*map(ValueError, validate_lints)])
  | exceptiongroup.ExceptionGroup: lints (1 sub-exception)
  +-+---------------- 1 ----------------
    | ValueError: {'build_id': 43, 'user_or_org': 'TileDB-Inc', 'project_name': 'CI'} is not valid under any of the given schemas
    |
    | Failed validating 'anyOf' in schema['properties']['azure']:
    |     {'anyOf': [{'$ref': '#/$defs/AzureConfig'}, {'type': 'null'}],
    |      'description': 'Azure Pipelines CI settings. This is usually '
    |                     'read-only and should not\n'
    |                     'normally be manually modified. Tools like '
    |                     'conda-smithy may modify this, as needed.\n'
    |                     'For example:\n'
    |                     '\n'
    |                     '```yaml\n'
    |                     'azure:\n'
    |                     '    # flag for forcing the building all supported '
    |                     'providers\n'
    |                     '    force: False\n'
    |                     '    # toggle for storing the conda build_artifacts '
    |                     'directory (including the\n'
    |                     '    # built packages) as an Azure pipeline artifact '
    |                     'that can be downloaded\n'
    |                     '    store_build_artifacts: False\n'
    |                     '    # toggle for freeing up some extra space on the '
    |                     'default Azure Pipelines\n'
    |                     '    # linux image before running the Docker container '
    |                     'for building\n'
    |                     '    free_disk_space: False\n'
    |                     '    # limit the amount of CI jobs running '
    |                     'concurrently at a given time\n'
    |                     '    # each OS will get its proportional share of the '
    |                     'configured value\n'
    |                     '    max_parallel: 25\n'
    |                     '```\n'
    |                     '\n'
    |                     'Below is an example configuration for setting up a '
    |                     'self-hosted Azure agent for Linux:\n'
    |                     '\n'
    |                     '```yaml\n'
    |                     'azure:\n'
    |                     '    settings_linux:\n'
    |                     '    pool:\n'
    |                     '        name: your_local_pool_name\n'
    |                     '        demands:\n'
    |                     '          - some_key -equals some_value\n'
    |                     '    workspace:\n'
    |                     '        clean: all\n'
    |                     '    strategy:\n'
    |                     '        maxParallel: 1\n'
    |                     '```\n'
    |                     '\n'
    |                     'Below is an example configuration for adding a '
    |                     'swapfile on an Azure agent for Linux:\n'
    |                     '\n'
    |                     '```yaml\n'
    |                     'azure:\n'
    |                     '    settings_linux:\n'
    |                     '        swapfile_size: 10GiB\n'
    |                     '```'}
    |
    | On instance['azure']:
    |     {'build_id': 43, 'project_name': 'CI', 'user_or_org': 'TileDB-Inc'}
    +------------------------------------

Installed packages

# packages in environment at /home/wsl/mambaforge:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
anyio                     4.3.0              pyhd8ed1ab_0    conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
black                     24.2.0          py310hff52083_0    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
boltons                   23.1.1             pyhd8ed1ab_0    conda-forge
boolean.py                4.0                pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.27.0               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
chardet                   5.2.0           py310hff52083_1    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
cirun                     0.30               pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     24.1.2          py310hff52083_0    conda-forge
conda-build               24.1.2          py310hff52083_0    conda-forge
conda-forge-pinning       2024.03.14.11.25.07      hd8ed1ab_0    conda-forge
conda-index               0.4.0              pyhd8ed1ab_0    conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
conda-smithy              3.32.0             pyhd8ed1ab_0    conda-forge
cryptography              42.0.5          py310h75e40e8_0    conda-forge
curl                      8.5.0                hca28451_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
deprecated                1.2.14             pyh1a96a4e_0    conda-forge
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
distro                    1.9.0              pyhd8ed1ab_0    conda-forge
editables                 0.5                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fmt                       10.2.1               h00ab1b0_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
git                       2.44.0          pl5321h709897a_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.42             pyhd8ed1ab_0    conda-forge
gmp                       6.3.0                h59595ed_1    conda-forge
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hatch                     1.9.4              pyhd8ed1ab_0    conda-forge
hatchling                 1.21.1             pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
httpcore                  1.0.4              pyhd8ed1ab_0    conda-forge
httpx                     0.27.0             pyhd8ed1ab_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
hyperlink                 21.0.0             pyhd3deb0d_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.2              pyha770c72_0    conda-forge
importlib_metadata        7.0.2                hd8ed1ab_0    conda-forge
importlib_resources       6.3.0              pyhd8ed1ab_0    conda-forge
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jaraco.classes            3.3.1              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310hff52083_3    conda-forge
jsonschema                4.21.1             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
keyring                   24.3.1          py310hff52083_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.7.2                h2aa1ff5_1    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libglib                   2.80.0               hf2295e7_0    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblief                   0.12.3               h27087fc_0    conda-forge
libmamba                  1.5.7                had39da4_0    conda-forge
libmambapy                1.5.7           py310h39ff949_0    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.28               hfc55251_0    conda-forge
libsqlite                 3.45.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.5               h232c23b_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
license-expression        30.1.1             pyhd8ed1ab_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.5.7           py310h51d5547_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5           py310h2372a71_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
menuinst                  2.0.2           py310hff52083_0    conda-forge
more-itertools            10.2.0             pyhd8ed1ab_0    conda-forge
msrest                    0.6.21             pyh44b312d_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
patch                     2.7.6             h7f98852_1002    conda-forge
patchelf                  0.17.2               h58526e2_0    conda-forge
pathspec                  0.12.1             pyhd8ed1ab_0    conda-forge
pcre2                     10.43                hcad00b1_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkginfo                   1.10.0             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
psutil                    5.9.8           py310h2372a71_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py-lief                   0.12.3          py310hd8f1fbe_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py310h2372a71_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pycryptodome              3.20.0          py310hb0f0acc_0    conda-forge
pygithub                  2.2.0              pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_1    conda-forge
pynacl                    1.5.0           py310h2372a71_3    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_1_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-libarchive-c       5.1             py310hff52083_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.33.0             pyhd8ed1ab_0    conda-forge
reproc                    14.2.4.post0         hd590300_1    conda-forge
reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.4.0              pyhd8ed1ab_0    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
ripgrep                   14.1.0               he8a937b_0    conda-forge
rpds-py                   0.18.0          py310hcb5633a_0    conda-forge
ruamel.yaml               0.18.6          py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.8           py310h2372a71_0    conda-forge
scrypt                    0.8.24          py310hd2d1899_0    conda-forge
secretstorage             3.3.3           py310hff52083_2    conda-forge
setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
shellingham               1.5.4              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomli-w                   1.0.0              pyhd8ed1ab_0    conda-forge
tomlkit                   0.12.4             pyha770c72_0    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
trove-classifiers         2024.3.3           pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
typer                     0.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.10.0               hd8ed1ab_0    conda-forge
typing_extensions         4.10.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
userpath                  1.7.0              pyhd8ed1ab_0    conda-forge
virtualenv                20.25.1            pyhd8ed1ab_0    conda-forge
vsts-python-api           0.1.25             pyhd8ed1ab_1    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.16.0          py310h2372a71_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.22.0          py310h1275a96_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Environment info

active environment : base
    active env location : /home/wsl/mambaforge
            shell level : 1
       user config file : /home/wsl/.condarc
 populated config files : /home/wsl/mambaforge/.condarc
          conda version : 24.1.2
    conda-build version : 24.1.2
         python version : 3.10.13.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.1.2=0
                          __glibc=2.35=0
                          __linux=5.15.146.1=0
                          __unix=0=0
       base environment : /home/wsl/mambaforge  (writable)
      conda av data dir : /home/wsl/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/wsl/mambaforge/pkgs
                          /home/wsl/.conda/pkgs
       envs directories : /home/wsl/mambaforge/envs
                          /home/wsl/.conda/envs
               platform : linux-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.10.13 Linux/5.15.146.1-microsoft-standard-WSL2 ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.7
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

jdblischak commented 5 months ago

I assume this change in behavior was introduced with the new schemas in https://github.com/conda-forge/conda-smithy/pull/1756

jaimergp commented 5 months ago

build_id is only needed for the README and can be calculated, so maybe you don't need it:

https://github.com/conda-forge/conda-smithy/blob/44acb937c50ce98adf1b50b01a6ab7bc577d2fb2/conda_smithy/azure_ci_utils.py#L270-L287

user_or_org is explicitly excluded from the schema 🤔 Do you remember why this needed to be excluded, @isuruf?

https://github.com/conda-forge/conda-smithy/blob/44acb937c50ce98adf1b50b01a6ab7bc577d2fb2/conda_smithy/schema.py#L206-L212

Note it can also be added via the github key (assuming they are the same).

https://github.com/conda-forge/conda-smithy/blob/44acb937c50ce98adf1b50b01a6ab7bc577d2fb2/conda_smithy/configure_feedstock.py#L2136-L2137

jdblischak commented 5 months ago

@jaimergp thanks for your help investigating! Today I tested what I would get if I completed removed build_id, user_or_org, and project_name.

Here is an example URL that was properly constructed by conda-smithy 3.31.0 with the following settings:

azure:
  build_id: 44
  user_or_org: TileDB-Inc
  project_name: CI

https://dev.azure.com/TileDB-Inc/CI/_build/latest?definitionId=44&branchName=main

When I comment out the above, then I get the following URL from conda-smithy 3.32.0:

https://dev.azure.com/TileDB-Inc/feedstock-builds/_build/latest?definitionId=&branchName=main

Note it can also be added via the github key (assuming they are the same).

The org is correct in this case because we do set it within github, but I don't think it should be required to have the same org name in GitHub and Azure

build_id is only needed for the README and can be calculated, so maybe you don't need it:

The build_id may be getting calculated, but it currently is not being inserted into the Azure URLs in the README

And the biggest problem is the Azure project name. Ours is "CI", not the conda-forge default of "feedstock-builds"

jaimergp commented 5 months ago

Yes, we need to amend a couple details here, and we'll do so shortly. I was suggesting possible workarounds in the meantime. Thanks for looking into it!

What happens if you only leave project_name in? I don't think that breaks the schema.

jdblischak commented 5 months ago

What happens if you only leave project_name in? I don't think that breaks the schema.

Confirmed. I tested that I could edit project_name and the new value was inserted into the README Azure URLs.

So the to-do's to close this Issue are:

[x] Restore the Azure field user_or_org (turns out this worked; the problem was the description; fixed in #1871)
[x] Restore the Azure field build_id (added back in #1871)

While it would be a nice new behavior to automatically calculate and insert the build_id into the URLs, my immediate need is to restore the previous behavior. I have many feedstocks that use Azure/build_id, and if possible I'd prefer to not have to update all of them.

isuruf commented 5 months ago

Added https://github.com/conda-forge/conda-smithy/pull/1871

jdblischak commented 5 months ago

Closed by #1871 Thanks @isuruf!

conda-forge / conda-smithy