conda-forge recipe fails tests - python -m build is incomplete

[prjemian]

import apstools fails when trying to build for conda-forge. This blocks the final release for v1.6.19.

[prjemian]

Found the same error after installing by pip today for XPCS.

pip install "apstools >=1.6.19rc3" --no-deps

[prjemian]

Started with clean conda environment and Python 3.11:

conda create -n test "python ==3.11"
conda activate test
pip install "apstools >=1.6.19rc3"

output:

Collecting apstools>=1.6.19rc3
  Downloading apstools-1.6.19rc3-py3-none-any.whl.metadata (6.8 kB)
Collecting area-detector-handlers (from apstools>=1.6.19rc3)
  Downloading area_detector_handlers-0.0.10-py3-none-any.whl.metadata (1.4 kB)
Collecting bluesky!=1.11.0,>=1.6.2 (from apstools>=1.6.19rc3)
  Downloading bluesky-1.12.0-py3-none-any.whl.metadata (3.4 kB)
Collecting bluesky-live (from apstools>=1.6.19rc3)
  Downloading bluesky_live-0.0.8-py3-none-any.whl.metadata (1.1 kB)
Collecting databroker==1.2.5 (from apstools>=1.6.19rc3)
  Downloading databroker-1.2.5-py2.py3-none-any.whl.metadata (6.2 kB)
Collecting databroker-pack (from apstools>=1.6.19rc3)
  Downloading databroker_pack-0.3.0-py3-none-any.whl.metadata (1.1 kB)
Collecting entrypoints (from apstools>=1.6.19rc3)
  Downloading entrypoints-0.4-py3-none-any.whl.metadata (2.6 kB)
Collecting h5py (from apstools>=1.6.19rc3)
  Downloading h5py-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB)
Collecting intake<=1 (from apstools>=1.6.19rc3)
  Downloading intake-0.7.0-py3-none-any.whl.metadata (4.3 kB)
INFO: pip is looking at multiple versions of apstools to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.5.4 Requires-Python >=3.7,<3.9; 1.5.4rc1 Requires-Python >=3.7,<3.9; 1.5.4rc2 Requires-Python >=3.7,<3.9; 1.5.4rc3 Requires-Python >=3.7,<3.9; 1.6.0 Requires-Python >=3.7, <3.10; 1.6.1 Requires-Python >=3.7, <3.10; 1.6.2 Requires-Python >=3.7, <3.10; 1.6.2rc0 Requires-Python >=3.7, <3.10; 1.6.3 Requires-Python >=3.8, <3.11; 1.6.3rc1 Requires-Python >=3.8, <3.11; 1.6.3rc2 Requires-Python >=3.8, <3.11; 1.6.3rc3 Requires-Python >=3.8, <3.11; 1.6.3rc4 Requires-Python >=3.8, <3.11; 1.6.3rc5 Requires-Python >=3.8, <3.11
ERROR: Could not find a version that satisfies the requirement matplotlib-base (from apstools) (from versions: none)
ERROR: No matching distribution found for matplotlib-base

[prjemian]

Confirmed there is no such matplotlib-base package on PyPI. This line should be changed to "matplotlib",: https://github.com/BCDA-APS/apstools/blob/83206b9fab067f9a80bd4f9fa732b9c0c51d12fa/pyproject.toml#L63

Confirmed that a matplotlib-base package exists on conda-forge. This line should not change: https://github.com/BCDA-APS/apstools/blob/83206b9fab067f9a80bd4f9fa732b9c0c51d12fa/environment.yml#L19

[prjemian]

Still failing:

2024-04-17T23:09:47.7047264Z export SRC_DIR=/home/conda/feedstock_root/build_artifacts/apstools_1713395125623/test_tmp
2024-04-17T23:09:48.0835127Z import: 'apstools'
2024-04-17T23:09:48.0836309Z Traceback (most recent call last):
2024-04-17T23:09:48.0837385Z   File "/home/conda/feedstock_root/build_artifacts/apstools_1713395125623/test_tmp/run_test.py", line 2, in <module>
2024-04-17T23:09:48.0837929Z     import apstools
2024-04-17T23:09:48.0838360Z ModuleNotFoundError: No module named 'apstools'
2024-04-17T23:09:49.7144482Z WARNING: Tests failed for apstools-1.6.19rc4-pyhd8ed1ab_0.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken
2024-04-17T23:09:49.7758914Z TESTS FAILED: apstools-1.6.19rc4-pyhd8ed1ab_0.conda

[prjemian]

Debugging advice.

[prjemian]

useful commands when debugging locally

conda build purge
conda-build recipe -c apsu -c conda-forge
export BUILD_DIR=${CONDA_PREFIX}/conda-bld/
conda env remove -n test
conda create -yn test "python ==3.11"
conda install -yn test -c file://${BUILD_DIR} -c apsu -c conda-forge "apstools >1.6.18"
conda activate test
python -c "import apstools"
python -c "import apstools.synApps"
spec2ophyd --version

[prjemian]

Build in a workstation with no customized Python installation. A container is useful for this case:

docker run -it --rm condaforge/mambaforge:latest /bin/bash
pip install "apstools >=1.6.18rc3"

python -c "import apstools"

and this fails with

(base) root@56e128f2d18c:~# python -c "import apstools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'apstools'

Further investigation shows the pip installation did not actually install the apstools code, expected to see apstools/ directory in the next interaction:

(base) root@56e128f2d18c:~# ls ${CONDA_PREFIX}/lib/python3.10/site-packages/ | grep aps
apstools-1.6.19rc4.dist-info

Next step is to download the apstools repo (in the container) and investigate why the wheel is not installed by pip.

[prjemian]

Use a new container:

docker run -it --rm condaforge/mambaforge:latest /bin/bash

# and in the container

git clone https://github.com/BCDA-APS/apstools
mkdir wheel
cd wheel
pip wheel ../apstools

# now try installing the new package
conda create -yn test "python ==3.11"
conda activate test
pip install -f . apstools

# and test it
python -c "import apstools"

success with Py3.11:

(test) root@379e444ed185:/wheel# python -c "import apstools"
(test) root@379e444ed185:/wheel# ls ${CONDA_PREFIX}/lib/python3.11/site-packages/ | grep aps
apstools
apstools-1.6.18.dist-info

[prjemian]

repeat with Py3.10, which also succeeds:

(test) root@1d666095a796:/wheel# python -c "import apstools"
(test) root@1d666095a796:/wheel# ls ${CONDA_PREFIX}/lib/python3.10/site-packages/ | grep aps
apstools
apstools-1.6.18.dist-info

[prjemian]

Repeat those steps but with pip install -e . in the `apstools directory. In each case above, the 1.6.18 version was installed, not the one under test.

[prjemian]

Python 3.10:

docker run -it --rm condaforge/mambaforge:latest /bin/bash

# and in the container

conda create -yn test "python ==3.10"
conda activate test

git clone https://github.com/BCDA-APS/apstools
# make sure to install dependencies!
pip install -e apstools

# and test
cd /tmp
python -c "import apstools"
ls ${CONDA_PREFIX}/lib/python3.10/site-packages/ | grep aps

successful results

(test) root@4cbc86bd6879:/tmp# python -c "import apstools"
(test) root@4cbc86bd6879:/tmp# ls ${CONDA_PREFIX}/lib/python3.10/site-packages/ | grep aps
__editable__.apstools-1.6.19rc4.pth
__editable___apstools_1_6_19rc4_finder.py
apstools-1.6.19rc4.dist-info

[prjemian]

Also worked with Py3.11

[prjemian]

Try building a wheel again and installing it.

docker run -it --rm condaforge/mambaforge:latest /bin/bash

in the container

git clone https://github.com/BCDA-APS/apstools
mkdir wheel
cd wheel
pip wheel ../apstools
export WHEEL=$(ls | grep apstools*)

# now try installing the new package
conda create -yn test "python ==3.11"
conda activate test
pip install ${WHEEL}

# check the installed version
pip list | grep apstools

# and test it
python -c "import apstools"

failed result:

(test) root@67ee62088bf9:/wheel# pip list | grep apstools
apstools                  1.6.19rc4
(test) root@67ee62088bf9:/wheel# python -c "import apstools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'apstools'

[prjemian]

The conda-forge process fails because the pip installation step in the build fails. No build/lib directory.

Not seeing any reports of the fail in the logs though and this makes it very difficult to identify the root problem. Suspecting some misconfiguration in pyproject.toml file. This release includes a transition from setup.cfg per PEP518.

[prjemian]

Example (most-recent) conda-forge build log: https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/917454/logs/10

(logs will be deleted after some time, perhaps 90 days from when generated)

[prjemian]

note: In that log, the pip installation starts at timestamp 2024-04-17T23:07:40.9623314Z.

[prjemian]

These steps are used in apstools workflow to build package for PyPI

# ... setup Py3.11
conda create -yn test "python ==3.11"
conda activate test

# workflow starts in the source directory, get it
git clone https://github.com/bcda-aps/apstools
cd apstools

# build sdist (tarball) and wheel
python -m pip install build --user
python -m build --sdist --wheel --outdir dist/ .
python -m pip install twine
twine check dist/*

# try to install & test it
conda env create -y -f environment.yml -n apstools
conda activate apstools
python -m pip install --no-deps --no-build-isolation dist/apstools-*.whl
python -c "import apstools"
spec2ophyd --version
# pip uninstall -y apstools

[prjemian]

Still not able to identify the root cause. Time to revisit the work of #914. In a new branch from the v1.6.18 tag, start over and check that wheel can be installed.

Some helpful commands for that, working in the source code root directory ...

conda env remove -n test
conda create -yn test "python ==3.11"
conda activate test

# build sdist (tarball) and wheel
python -m pip install build twine
python -m build --sdist --wheel --outdir dist/ .
export WHEEL=$(ls -1t dist/apstools-*.whl | head -1)
twine check ${WHEEL}
echo "WHEEL=${WHEEL}"

# install
conda env create -yf environment.yml -n test-apstools
conda activate test-apstools
python -m pip install --no-deps --no-build-isolation ${WHEEL}

# test
ls ${CONDA_PREFIX}/lib/python3.1/site-packages/ | grep aps
echo -n "PYTHON IMPORT TEST: "
python -c "import apstools"
python -c "import apstools.callbacks"
python -c "import apstools.devices"
python -c "import apstools.plans"
python -c "import apstools.utils"
python -c "import apstools.synApps"
echo ""
echo -n "spec2ophyd VERSION TEST: "
spec2ophyd --version

pip list --verbose | grep -v " conda"
# pip uninstall -y apstools

conda deactivate

[prjemian]

Observe that when the installations fail, the wheel file is a few kB, but a few 100 kB when successful.

[prjemian]

In new branch, starting from 1.6.18 tag, all builds succeed to install properly when current pyproject.toml and MANIFEST.in are used. Problem is not with the packaging.

[prjemian]

Discovery

When the builds succeed, inspection of the build/ directory showed content in the build/lib/ directory. When the builds fail, there is no such directory, or content within. Once that directory is created, it is not deleted unless done manually. Because of this, various test permutations above have shown false positives.

During the build process (python -m build --sdist --wheel --outdir dist/ .), this is the only line of output that led to discovery of the root problem and its solution:

warning: install_lib: 'build/lib' does not exist -- no Python modules to install

Following that error message through a web search, this article pointed out that the apstools package code was not found by auto-discovery.

Possible causes of this warning include:

• The Python module(s) were not built or compiled yet.
• The build directory where the modules are supposed to be located was deleted or is missing.
• There is an issue with the installation process, such as missing dependencies or errors during the build process.

The first two of these can be rejected on the spot (direct evidence they do not apply). Following the third one, with guidance, this is the addition that solved the problem: https://github.com/BCDA-APS/apstools/blob/ece96632cb919c34c18bc200645189a5632fa576/pyproject.toml#L250-L251

Along the way, there was a suggestion that the autodiscovery might be more successful if the apstools/ directory were relocated into src/apstools/. Currently, without https://github.com/BCDA-APS/apstools/blob/ece96632cb919c34c18bc200645189a5632fa576/pyproject.toml#L247-L248 autodiscovery identifies three top-level directories:

(test) prjemian@arf:~/.../BCDA-APS/apstools$ python -m build --sdist --wheel --outdir dist/ .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools>=64
  - setuptools_scm[toml]>=8.0
* Getting build dependencies for sdist...
error: Multiple top-level packages discovered in a flat-layout: ['ideas', 'apstools', 'resources'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

and this is yet another clue.

[prjemian]

Consider a change of the build system away from setuptools to use tools better designed for the PEP518 packaging system. Alternatives include:

• meson
• poetry