BUG: something non-deterministic going wrong in squished_input_variants/preserve_top_level_loops

[h-vetinari]

This issue first showed up a long time ago: https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/4190

It occurs with regularity while trying to completely skip pypy builds on an already-migrated feedstock. Of course it would be possible to drop the migrator file completely, but I don't know if that doesn't put the migration bot in an inconsistent state somehow, and feels more intrusive than just a (often temporary) skip.

I hit this again today in https://github.com/conda-forge/scipy-feedstock/pull/274, where we really need to skip PyPy builds for now.

The first thing to note is that debugging this is hard because the rerender changes based on random print statements or other changes I've made to my local smithy checkout (or even the ordering: length in the pypy migrator). This points to some indeterminism (e.g. dict/set order).

So far I was able to narrow it down to the following: things go wrong in https://github.com/conda-forge/conda-smithy/blob/a04dba97f7f204c6df401bb2cda9c68644c8ad42/conda_smithy/configure_feedstock.py#L626-L627 Before that line, we have

used_key_values['python_impl']=['cpython', 'cpython', 'cpython', 'cpython']

whereas after it we get (for the failing case)

used_key_values['python_impl']=('pypy', 'cpython', 'cpython', 'cpython', 'cpython', 'cpython')

This happens against a background of the following values:

top_level_loop_vars={'target_platform', 'numpy', 'python', 'python_impl'}
all_used_vars={'docker_image', 'MACOSX_DEPLOYMENT_TARGET', 'libblas', 'channel_sources', 'macos_machine', 'cdt_name', 'liblapack', 'channel_targets', 'BUILD', 'numpy', 'python', 'pin_run_as_build', 'MACOSX_SDK_VERSION', 'python_impl', 'c_compiler', 'target_platform', 'macos_min_version', 'c_stdlib', 'cxx_compiler', 'zip_keys', 'cdt_arch', 'build_number_decrement', 'libcblas'}
preserve_top_level_loops={'python_impl'}

Given the definition https://github.com/conda-forge/conda-smithy/blob/a04dba97f7f204c6df401bb2cda9c68644c8ad42/conda_smithy/configure_feedstock.py#L579

it's possible that python_impl should be detected as used (if only from skip: true # [python_impl == "pypy"]) but isn't. Beyond that, I've kind of saturated on smithy code for now 😅 Any help, @beckermr? 🙏

[jakirkham]

cc @mbargull (in case you have any insights here)

[beckermr]

As usual to debug this it is likely important to set the python hash seed. :/

[beckermr]

Too much stuff was changed on the branch for scipy above for me to be able to debug. @h-vetinari do you have a cleaner branch I can look at?

[h-vetinari]

do you have a cleaner branch I can look at?

The problem should reproduce if you simply add skip: true # [python_impl == "pypy"] on the main branch of the scipy feedstock, but it seems you already were able to track something down? Thanks a lot! 🙏