configure.py crashes connected to distutils and/or setuptools

[dwpaley]

Recent commits are hitting test failures for xfel_psana_centos_7 python3.9 on step "Configure and build" with tracebacks that look like this:

Traceback

``` Traceback (most recent call last): File "../modules/cctbx_project/libtbx/configure.py", line 34, in if not run(): File "../modules/cctbx_project/libtbx/configure.py", line 29, in run libtbx.env_config.cold_start(sys.argv) File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 3017, in cold_start env.refresh() File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 2251, in refresh module.process_libtbx_refresh_py() File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 2568, in process_libtbx_refresh_py exec(to_str(fh.read()), global_vars) File "", line 24, in File "/__w/1/modules/cctbx_project/libtbx/pkg_utils.py", line 54, in import setuptools File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/setuptools/__init__.py", line 8, in import _distutils_hack.override # noqa: F401 File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/override.py", line 1, in __import__('_distutils_hack').do_override() File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 73, in do_override ensure_local_distutils() File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils assert '_distutils' in core.__file__, core.__file__ AssertionError: /__w/1/miniforge/envs/psana_env/lib/python3.7/distutils/core.py ```

It looks like the same issue discussed in https://github.com/pypa/pip/issues/10742 and https://github.com/pypa/setuptools/issues/2993. I don't have any real idea what's going on, so just flagging this for now.

[bkpoon]

For now, set the version of setuptools to <60. That's what I did for the 2021.12 release and the nightly package builds.

[dwpaley]

For now, set the version of setuptools to <60. That's what I did for the 2021.12 release and the nightly package builds.

Thanks!

[phyy-nx]

Bump. Hit this again at NERSC. We are pinning the NERSC environments to setuptools<60 as well.

[Trzs]

Bump. Encountered this issue on a new Nersc test system. Pinning setuptools<60 solved it.

[dwpaley]

Alright, this is clearly a headache and is not fixing itself. I’ll look at it next week and try to figure out what’s going on.

[dwpaley]

Got some clarity on this from @abravalheri. It's apparently well known that importing pip will break setuptools because setuptools packages a modified distutils while pip requires the unmodified stdlib distutils. We can avoid using pip and setuptools at the same time, or else export an environment variable SETUPTOOLS_USE_DISTUTILS=stdlib.

[dwpaley]

On first glance I only see that pip is really needed in pkg_utils.require. @bkpoon any chance we can quarantine calls to require inside subprocesses so that we don't have to import pip otherwise?

[graeme-winter]

Bump. Encountered this issue on a new Nersc test system. Pinning setuptools<60 solved it.

In DIALS we got bitten by this and yes

silver-surfer-2 .conda-envs :) [main] $ grep setuptools *
linux.txt:conda-forge::setuptools<60
macos.txt:conda-forge::setuptools<60
windows.txt:conda-forge::setuptools<60

We just pinned setuptools