Change conda-build-version to mamba-version

[brocksam]

This PR attempts to fix #177 by replacing conda-build-version in conda-incubator/setup-miniconda step of all jobs in all GitHub Actions workflows with mamba-version, as per https://github.com/conda-incubator/setup-miniconda/issues/116.

[moorepants]

This is the workaround at the top of the issue now:

1. Use Mambaforge

   Remove mamba-version. Add miniforge-variant: Mambaforge (and optionally miniforge-version).

[moorepants]

I'm not seeing any output in the test step.

[moorepants]

Yeh, the tests aren't even running:

[moorepants]

Nothing is running:

The installing doesn't run and cyipopt "installs" with no error as a wheel.

[moorepants]

I had to change the run steps to a different multi-line syntax. I don't know why what we have in master didn't work.

Now the tests are failing, seemingly because the mamba installs of scipy aren't executing.

[moorepants]

The mamba commands don't show output because of the -q flag. Just realized that.

[moorepants]

Looks like some scipy versions are failing with:

>       from scipy.sparse import coo_array
E       ImportError: cannot import name 'coo_array' from 'scipy.sparse' (/usr/share/miniconda3/envs/test-environment/lib/python3.8/site-packages/scipy/sparse/__init__.py)

So the recent PR isn't handling imports correctly in some cases.

[moorepants]

I'm guessing this is a mamba install fail, but there is no output so hard to know:

Run python -c "import cyipopt"
============================= test session starts ==============================
platform linux -- Python 3.11.0, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/runner/work/cyipopt/cyipopt, configfile: pytest.ini, testpaths: cyipopt/tests
collected 14 items

cyipopt/tests/integration/test_hs071.py .                                [  7%]
cyipopt/tests/unit/test_deprecations.py .....                            [ 42%]
cyipopt/tests/unit/test_scipy_optional.py .sssssss                       [100%]

========================= 7 passed, 7 skipped in 0.12s =========================
Error: Process completed with exit code 1.

[moorepants]

Good news is that using mambaforge fixes the first issue.

[moorepants]

I don't understand why I can't see the output from the mamba commands. It is impossible to debug without any information.

[moorepants]

These are the failing builds:

mac py 3.10, ipopt 3.12 mac py 3.11, ipopt 3.12 linux py 3.11, ipopt 3.12

It seems to fail when scipy is installed. So maybe there isn't a compatible scipy for these envs?

[moorepants]

@brocksam do you know why the output from the mamba install commands is not displayed to stdout?

[moorepants]

I made this conda env file that does Py 3.11 and Ipopt 3.12 on linux:

name: cyipopt-fail
channels:
  - conda-forge
dependencies:
  - cython >=0.26
  - ipopt ==3.12.*
  - libblas * *netlib
  - numpy >=1.15
  - pkg-config >=0.29.2
  - pytest >=3.3.2
  - python ==3.11.*
  - scipy >=0.19.1
  - setuptools >=39.0
  - sphinx

then

mamba env create -f cyipopt-fail.yml

worked for me. This should be the same combo of packages that fails in CI.

[moorepants]

With the -v flag we see:

info     libmamba Parsing MatchSpec ca-certificates
info     libmamba Parsing MatchSpec openssl
info     libmamba Parsing MatchSpec openssl
info     libmamba Parsing MatchSpec python 3.11.*
info     libmamba Problem count: 1
info     libmamba Freeing solver.
info     libmamba Freeing pool.
Error: Process completed with exit code 1.

which doesn't seem to give any more clues.

[moorepants]

This might be related: https://github.com/mamba-org/mamba/issues/2193

[moorepants]

The -vv flag gave no more info at failure than -v. I'm think we should just merge this, as at least the tests run for most combinations.

[moorepants]

This could also be a sign to drop Ipopt 3.12 support.

[moorepants]

Ipopt 3.12.13 (last version) was released in April of 2019. Seems reasonable to drop it.

[moorepants]

I'm running Ubuntu 22.10 and it includes Ipopt 3.11.9. So maybe we shouldn't be so hasty in removing 3.12.

[moorepants]

I'm back to conda (solution 2 in the related action issue). And I get more info:

# packages in environment at /usr/share/miniconda/envs/test-environment:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
ampl-mp                   3.1.0             hc99cbb1_1004    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
blas                      2.17                   openblas    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
cyipopt                   1.3.0.dev0               pypi_0    pypi
cython                    0.29.33         py310heca2aa9_0    conda-forge
exceptiongroup            1.1.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipopt                     3.13.3               hcc5340a_5    conda-forge
lapack                    3.6.1                ha44fe06_2    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.8.0               17_openblas    conda-forge
libcblas                  3.8.0               17_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
liblapack                 3.8.0               17_openblas    conda-forge
liblapacke                3.8.0               17_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.10          pthreads_hb3c22a3_5    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               15.0.7               h0cdce71_0    conda-forge
metis                     5.1.0             h58526e2_1006    conda-forge
mumps-include             5.2.1               ha770c72_11    conda-forge
mumps-seq                 5.2.1               hc1830dd_10    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.22.4          py310h4ef5377_0    conda-forge
openssl                   3.0.8                h0b41bf4_0    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pip                       23.0               pyhd8ed1ab_0    conda-forge
pkg-config                0.29.2            h36c2ea0_1008    conda-forge
pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
pytest                    7.2.1              pyhd8ed1ab_0    conda-forge
python                    3.10.9          he550d4f_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
scipy                     1.7.3           py310h4f1e569_0  
scotch                    6.0.9                hb2e6521_2    conda-forge
setuptools                67.1.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
============================= test session starts ==============================
platform linux -- Python 3.10.9, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/runner/work/cyipopt/cyipopt, configfile: pytest.ini, testpaths: cyipopt/tests
collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/usr/share/miniconda/envs/test-environment/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:883: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
cyipopt/__init__.py:12: in <module>
    from ipopt_wrapper import *
E   ImportError: libipopt.so.1: cannot open shared object file: No such file or directory
=========================== short test summary info ============================
ERROR  - ImportError: libipopt.so.1: cannot open shared object file: No such file or directory
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.06s ===============================
Error: Process completed with exit code 2.

I see now that ipopt is being upgraded when we install scipy. That is likely the problem.

[moorepants]

Apparently --freeze-installed doesn't do what it advertises. The packages still updated.

[moorepants]

Here is a new clue from the failing mac build:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package scipy-1.10.0-py311h939689b_0 requires libgfortran5 >=11.3.0, but none of the providers can be installed

Maybe the new scipy versions have this new libfortran requirement that is incompatible with ipopt being build with libfortran4.

[moorepants]

Py 3.11 & Ipopt 3.12 Ubuntu

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package scipy-0.17.1-np110py27_blas_openblas_200 requires python 2.7*, but none of the providers can be installed

Py 3.11 & Ipopt 3.12 Mac

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package scipy-0.17.1-np110py27_blas_openblas_200 requires python 2.7*, but none of the providers can be installed

Py 3.10 & Ipopt 3.12 Mac

    from ipopt_wrapper import *
E   ImportError: dlopen(/usr/local/miniconda/envs/test-environment/lib/python3.10/site-packages/ipopt_wrapper.cpython-310-darwin.so, 0x0002): Library not loaded: '@rpath/libgfortran.4.dylib'
E     Referenced from: '/usr/local/miniconda/envs/test-environment/lib/python3.10/site-packages/ipopt_wrapper.cpython-310-darwin.so'
E     Reason: tried: '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/bin/../lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/bin/../lib/libgfortran.4.dylib' (no such file), '/usr/local/lib/libgfortran.4.dylib' (no such file), '/usr/lib/libgfortran.4.dylib' (no such file)
=========================== short test summary info ============================
ERROR  - ImportError: dlopen(/usr/local/miniconda/envs/test-environment/lib/python3.10/site-packages/ipopt_wrapper.cpython-310-darwin.so, 0x0002): Library not loaded: '@rpath/libgfortran.4.dylib'
  Referenced from: '/usr/local/miniconda/envs/test-environment/lib/python3.10/site-packages/ipopt_wrapper.cpython-310-darwin.so'
  Reason: tried: '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/bin/../lib/libgfortran.4.dylib' (no such file), '/usr/local/miniconda/envs/test-environment/bin/../lib/libgfortran.4.dylib' (no such file), '/usr/local/lib/libgfortran.4.dylib' (no such file), '/usr/lib/libgfortran.4.dylib' (no such file)
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.64s ===============================

[moorepants]

Looks like SciPy 1.8 can't be used with Py 3.11:

Looking for: ['cython', 'ipopt=3.12', 'numpy', 'pkg-config', 'pytest', 'python=3.11', 'scipy=1.8', 'setuptools', 'sphinx']

Could not solve for environment specs
Encountered problems while solving:
  - package scipy-1.8.0-py310hea5193d_0 requires python_abi 3.10.* *_cp310, but none of the providers can be installed

[moorepants]

There is some kind of incompatiblities associated with libfortran. I guess it is impossible to have ipopt 3.12 with python 3.11 and get a libfotran that works for both scipy and ipopt.

[moorepants]

I'll never get these hours back. What a nightmare. What we do for that green check mark.

[moorepants]

FYI I squashed things here. Branch history rewritten.

[moorepants]

Gonna go with this, we can edit in other PRs if needed.