Time to release a new version?

[bnavigator]

Hi,

with all the merged and pending PRs that would be ready to merge, I think it is warranted to take Slycot to the next release. For example, further development of https://github.com/python-control/python-control/pull/376 would need Slycot's support of complex matrices.

@roryyorke are you still around? Any other maintainer? I am also willing to help.

[murrayrm]

(slycot-test) murray@limani murrayrm % conda build conda-recipe-apple

[...]

conda.exceptions.ResolvePackageNotFound: 
  - scikit-build[version='>=0.10.0']

You forgot to activate the conda-forge channel. See #101 instructions:

conda build -c conda-forge conda-recipe-apple

Reran with -c conda-forge. conda-recipe-apple fails because it looks like it is looking for the wrong SDK. conda-recipe-openblas failed with a cryptic error "Failed building wheel for slycot". Various log files attached, but I am not sure this is useful to pursue further.

My overall sense/suggestions:

• We should be able to get python setup.py install working on all platforms as the mechanism for installing from source.

• Since conda-forge uses a separate set of recipes, I don't know that it is worth maintaining recipes separately in the main repository. 99% of users are going to install from the conda-forge binaries, I suspect, and the rest can (presumably) install using setup.py.

• For authors: I like Copyright (C) 2012-2020 Slycot team with a reasonably up-to-date CREDITS or AUTHORS file.

master_conda-recipe-apple.txt master_conda-recipe-openblas.txt

[bnavigator]

git shortlog -sn is apparently a way to discover authors; from this, I get:

    32    Benjamin Greiner
    18    bnavigator
     3    Ben

All the same

[bnavigator]

clang-9: warning: no such sysroot directory: '/opt/MacOSX10.9.sdk' [-Wmissing-sysroot]

Do you have the SDK in that directory? One thing @repagh and me ran into during #92 and co., that it is very easy do accidentally unpack into /opt/MacOSX10.9.sdk/MacOSX10.9.sdk

[bnavigator]

Thanks for the note about missing tests. So we must fix this, but can it be deferred to after release?

It's your decision. I very much prefer having one single instruction "run pytest" instead of telling Windows users to use a different command that does not even do the same thing.

One fix is to move $srcdir/slycot/tests to $srcdir/test/tests - then pytest runs fine

~That creates the danger of accidentally installing a test package into site-packages. I would rather keep the tests submodule under slycot~

On the other hand, the pytest docs suggest such a layout. With the correct arguments in setup.py, the tests won't get installed at all then. I see two options:

• 109 Move the tests to $srcdir/tests: Do not install them. Be able to run pytest which will automatically test with the installed package by not putting $srcdir/slycot into sys.path. python -m pytest or python -m unittest won't work, because they add the current directory to sys.path

• 110 Keep the current layout. Have the tests installed as slycot.tests. Run pytest --pyargs slycot.

[moorepants]

I think that it is best practice to include the unit tests with the release tarballs such that they are in a subdirectory of the main slycot package directory. This allows all downstream uses of slycot to rerun the tests if needed without having to copy them from the github repo manually. For example, the slycot-feedstock runs the tests after the build. This is also helpful for bug reports from end users too, you can ask them to run the tests using their installed version.

[murrayrm]

Back on the MacOS build problems: I've narrowed things down to the following line in slycot/CMakeLists.txt

python_extension_module(${SLYCOT_MODULE})

If you comment that out, MacOS version builds fine. I'm still trying to understand why since that command should work (according to scikit-build documentation). It might be that some of the APPLE specific commands elsewhere in the file conflict with the intended usage of this command (there are quite a few commands that look like voodoo to me).

[repagh]

Sorry guys, have been out of the loop for a while. I will try to give this a spin next week.

On Mon, 6 Apr 2020 at 02:44, Richard Murray notifications@github.com wrote:

Assigned #98 https://github.com/python-control/Slycot/issues/98 to @repagh https://github.com/repagh.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/python-control/Slycot/issues/98#event-3203256304, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVKIDMTJ4HLLB7VCOPWRWLRLEQYJANCNFSM4MASGTZQ .

-- René van Paassen | __o_/| Rene.vanPaassen@gmail.com <[___-----< t: +31 15 2628685 | o' mobile: +31 6 39846891

[repagh]

@murrayrm

In your error log:

  Ignoring CMAKE_OSX_SYSROOT value:

   /opt/MacOSX10.9.sdk

All conda's stuff (compilers etc) is built on that MacOSX sdk. The clean solution is to hunt it down somewhere, and install it in this magic location.

Pip and direct builds use the system's compilers (assuming you also have a Fortran), and don't have this issue.

I have some cleanups and changes:

• line length correction in SB04OD.f (limit to 72 characters, so a line fits on a punch card ;-)
• modified test_examples.py, to also use unittest, now all tests can run in one go
• small fixes to the CMakeLists.txt files, among others the minimum cmake required has gone up for scikit-build
• modified the conda recipes to force the right numpy include folder (this failed on Fedora 32, and no amount of tweaking could restore detection of numpy headers) and to run the slycot tests

I could not get the conda-recipe-mkl build to work, I get an error in the tests related to a missing mkl function. I am testing now on AMD Ryzen, Fedora 32. Maybe this does work for someone with an Intel processor.

[repagh]

I currently don't have a running Windows (virtual machine). Haven't been able to test Windows builds yet.

[repagh]

@roryyorke

E   ImportError: cannot import name '_wrapper' from partially initialized module 'slycot'

When running pytest in the installed slycot.tests directory it passes. If I move slycot/tests to blah/tests, and run pytest tests in directory blah, they also pass. If I move slycot/tests to test (i.e., in root of working tree), then they again fail. My guess: pytest wants the tests it runs to be arranged as a Python package (as in, directory with __init_.py) and this confuses the importing when tests appears to be a sub-package of slycot. We can look at fixing this if we get Windows CI going (#108); for now I'll just recommend python -m unittest on Windows in README.rst

This is not Windows specific. I also get this on Linux when the installed module does not match the module in $srcdir/slycot. Something with the scikit-build or setuptools initiated moving around of the compiled module between _skbuild, installed site-packages and $srcdir/slycot is broken and exposed by the way pytest imports modules.

The problem is way simpler. If python finds a folder "slycot" in your working directory, it tries to use that folder when doing "import slycot". That is why testing fails. Maybe we should consider renaming that folder "src".

[roryyorke]

I currently don't have a running Windows (virtual machine). Haven't been able to test Windows builds yet.

I've tested Python 3.7 and 3.8 (see #101 instructions for Windows).

The problem is way simpler. If python finds a folder "slycot" in your working directory, it tries to use that folder when doing "import slycot". That is why testing fails. Maybe we should consider renaming that folder "src".

pytest use was fixed in #110, though only for an "installed" Slycot. I'm not sure if a "non-installed" Slycot makes much sense (one needs slycot._wrapper), and I haven't tried the half-way case of an editable install - though as I understand in that case the "local" code is added to the Python module path.

As I see it there are four items to resolve for 0.4.0:

• 73 -- @repagh, do you want this in for 0.4.0? There were quite a few review comments

• Must we keep or remove conda recipes? @repagh, AFAICT no-one else seems to need them -- do you find them useful?
• success with "python setup.py install" on macOS; @murrayrm is looking at this.

• 113 and #115 - do we want to remove "strong" dependency on pytest?

[repagh]

• 73 has been superseded by #116 (thanks bnavigator), I accepted that.

• I do find the conda recipes useful. If only for trying/fixing some issue, and quickly sharing an experimental slycot through anaconda.org.
• On my mac, python setup.py install works. I am using a conda environment for that, with python 3.7. If this keeps eluding @murrayrm, we might try exchanging logs?
• AFAICT pytest is used now for conda building in the test phase? Is that a problem? Everything now seems to test fine with only pytest. #115 looks good to me now, I axed #113.

[murrayrm]

@repagh For the mac version, just to make sure when I try things again: you are using latest master for testing?

[repagh]

@repagh For the mac version, just to make sure when I try things again: you are using latest master for testing?

No, I was using my slightly modified apple recipe, in #115. But I just now tried the plain, current python-control/Slycot master, and that works too.

I did the following

python setup.py install

This picked up an openblas from MacPorts, in /opt/local/lib . Worked mostly, with a warning in one of the examples, when tested with pytest --pyargs slycot

Next attempt (after cleaning the install with pip and removing _skbuild)

python setup.py install -- -DBLA_VENDOR=Apple

That picked up the Apple Accelerate framework. Pytest test worked perfectly.

I also ran a

conda build conda-recipe-apple

That again picked up the MacPorts openblas. The updated recipe in #115 fixes that, and will get the Apple optimized one.

I think the main lesson is to look at the output of the CMake configuration, and check that it is picking up the Python, NumPy and Blas/Lapack that you intended to use.

[roryyorke]

I think we're close to release (I thought that 4 weeks ago, though...). It looks like #101, #115, and #117 are ready to merge.

@murrayrm , are you happy that we go ahead with release? I gather you still haven't successfully built on macOS.

[roryyorke]

Release notes updated.

I'd like to close #46 and #94, and (1) recommend the reporters use conda and (2) try 0.4.0.

95 is a bit trickier, though I don't know if using pipenv makes much sense if we don't provide binary wheels on PyPI.

[murrayrm]

I've been swamped with a few other things, but let me grab the latest version of things and at least report back what happens on my machine. I'm confident I can get things to compile with one of the methods. I think the issue may be where OpenBLAS is installed (conda env versus system directory).

Will try to get to this in the next few days. Sorry for the hold up!

[repagh]

I think we're close to release (I thought that 4 weeks ago, though...). It looks like #101, #115, and #117 are ready to merge.

@murrayrm , are you happy that we go ahead with release? I gather you still haven't successfully built on macOS.

IMHO #122 is now complete, this would solve our currently oldest bug #11

125 now made it in too (corrected from 123)

124 also looks OK.

With that, I think we have a 0.4 in our hands. @roryyorke , do you agree and can you run this one?

[murrayrm]

I just tried compiling on the latest master (8dc3069) and I still get a segmentation fault on a local compile. I am doing the same thing that was (and is) working on v0.3.5:

conda create -n test-slycot python=3.7 numpy
conda activate test-slycot 
conda install -c conda-forge gcc openblas scikit-build cmake
python setup.py install

If I then try to run python setup.py test or go to python-control and try to run the tests from there, I get a segmentation fault. This seems to occur as soon as import slycot is called.

Note that I don't have openblas installed via MacPorts as @repagh has done, but it should be getting installed via conda (from condo-forge). I am also getting gfortran via condo-forge (it is part of the gcc package).

I don't think this is a blocker for releasing v0.4.0 and worst case I can always do a conda install of the conda-forge version. It is odd, though.

For posterity, it would be good if @repagh or others who have this working on MacOS can say something about their development environment, in particular where gfortran and openblas are coming from.

[roryyorke]

@repagh I'll update the release notes with the latest merges etc., and proceed with the release.

I'm plan to tag current master (8dc3069).

[roryyorke]

@repagh So I can add something on sb10fd for the release notes: newly added sb10fd tries to solve the H-infinity problem for a single value of gamma, where sb10ad searches for a minimum gamma.

[roryyorke]

I've updated the release notes https://github.com/python-control/Slycot/issues/98#issuecomment-612492357 above. If there are any changes, let me know soon; I plan to do the release later today.

[roryyorke]

I get this at the end of a build here:

====================================================== test session starts =======================================================
platform linux -- Python 3.6.9, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /home/rory/src/slycot, inifile: setup.cfg
collected 106 items                                                                                                              

tests/test_ab08n.py ..                                                                                                     [  1%]
tests/test_ag08bd.py ....                                                                                                  [  5%]
tests/test_analysis.py ...................                                                                                 [ 23%]
tests/test_examples.py .............                                                                                       [ 35%]
tests/test_exceptions.py ..                                                                                                [ 37%]
tests/test_mb.py ..........                                                                                                [ 47%]
tests/test_mc.py ...                                                                                                       [ 50%]
tests/test_sb.py .............................                                                                             [ 77%]
tests/test_sg02ad.py .                                                                                                     [ 78%]
tests/test_sg03ad.py ...                                                                                                   [ 81%]
tests/test_tb05ad.py ......                                                                                                [ 86%]
tests/test_td04ad.py .......                                                                                               [ 93%]
tests/test_tg01ad.py .                                                                                                     [ 94%]
tests/test_tg01fd.py ..                                                                                                    [ 96%]
tests/test_transform.py ....                                                                                               [100%]

====================================================== 106 passed in 3.26s =======================================================
 ** On entry to SB10FDEpsilonFullSB10HDEpsilonFull parameter number 27 had an illegal value

Is that final line expected? It looks like a Fortran routine complaining to stderr or stdout?

[roryyorke]

It's from a "too-small ldwork test", so it's fine; if I remove the test, as below, there's no message. It is disconcerting if one is not expecting it. I checked out Travis log, and couldn't see it there.

diff --git a/slycot/tests/test_sb.py b/slycot/tests/test_sb.py
index 556a09d..a8ec94c 100644
--- a/slycot/tests/test_sb.py
+++ b/slycot/tests/test_sb.py
@@ -133,8 +133,8 @@ def test_sb10fd():
     n, m, np, ncon, nmeas = 6, 5, 5, 2, 2

     # ldwork too small
-    assert_raises(SlycotParameterError, synthesis.sb10fd,
-        n, m, np, ncon, nmeas, gamma, A, B, C, D, tol, 1)
+    # assert_raises(SlycotParameterError, synthesis.sb10fd,
+    #     n, m, np, ncon, nmeas, gamma, A, B, C, D, tol, 1)
     Ak, Bk, Ck, Dk, rcond = synthesis.sb10fd(
         n, m, np, ncon, nmeas, gamma, A, B, C, D, tol, 900)
     Ak, Bk, Ck, Dk, rcond = synthesis.sb10fd(

[bnavigator]

I checked out Travis log, and couldn't see it there.

It is in some of the Travis logs and now that I checked locally, here on my system the Fortran routine even causes the whole process to abort. But with an exit status of 0, so a CI would not catch that:

[ben@voyagerS9:~/src]% pytest --pyargs slycot -v                                                                                                                   [0]
========================================================================= test session starts =========================================================================
platform linux -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/ben/src
plugins: cov-2.8.1, shutil-1.7.0, virtualenv-1.7.0, mock-2.0.0
collected 106 items                                                                                                                                                   

Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ab08n.py::test_ab08n::test_ab08nd PASSED                                                       [  0%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ab08n.py::test_ab08n::test_ab08nz PASSED                                                       [  1%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ag08bd.py::test_tg01fd::test1_ag08bd PASSED                                                    [  2%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ag08bd.py::test_tg01fd::test2_ag08bd PASSED                                                    [  3%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ag08bd.py::test_tg01fd::test3_ag08bd PASSED                                                    [  4%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_ag08bd.py::test_tg01fd::test4_ag08bd PASSED                                                    [  5%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab05nd-SlycotArithmeticError-1-checkvars0] PASSED                [  6%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab07nd-SlycotResultWarning-2-checkvars1] PASSED                  [  7%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ad-SlycotArithmeticError-3-checkvars2] PASSED                [  8%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ad-SlycotArithmeticError-erange3-checkvars3] PASSED          [  9%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ad-SlycotResultWarning-erange4-checkvars4] PASSED            [ 10%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ax-SlycotArithmeticError-2-checkvars5] PASSED                [ 11%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ax-SlycotResultWarning-erange6-checkvars6] PASSED            [ 12%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ad-SlycotArithmeticError-3-checkvars7] PASSED                [ 13%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09ad-SlycotResultWarning-erange8-checkvars8] PASSED            [ 14%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09md-SlycotArithmeticError-3-checkvars9] PASSED                [ 15%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09md-SlycotResultWarning-erange10-checkvars10] PASSED          [ 16%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09nd-SlycotArithmeticError-3-checkvars11] PASSED               [ 16%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab09nd-SlycotResultWarning-erange12-checkvars12] PASSED          [ 17%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13bd-SlycotArithmeticError-6-checkvars13] PASSED               [ 18%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13bd-SlycotResultWarning-erange14-checkvars14] PASSED          [ 19%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13dd-SlycotArithmeticError-4-checkvars15] PASSED               [ 20%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13ed-SlycotArithmeticError-1-checkvars16] PASSED               [ 21%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13fd-SlycotArithmeticError-erange17-checkvars17] PASSED        [ 22%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_analysis.py::test_ab_docparse[ab13fd-SlycotResultWarning-erange18-checkvars18] PASSED          [ 23%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[ab08nd_example] PASSED                                               [ 24%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[ab13bd_example] PASSED                                               [ 25%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[ab13dd_example] PASSED                                               [ 26%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[ab13ed_example] PASSED                                               [ 27%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[ab13fd_example] PASSED                                               [ 28%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[mc01td_example] PASSED                                               [ 29%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[sb02md_example] PASSED                                               [ 30%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[sb02od_example] PASSED                                               [ 31%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[sb03md_example] PASSED                                               [ 32%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[tb01pd_example] PASSED                                               [ 33%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[tb03ad_example] PASSED                                               [ 33%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[tb05ad_example] PASSED                                               [ 34%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_examples.py::test_example[tc04ad_example] PASSED                                               [ 35%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_exceptions.py::test_standard_info_error PASSED                                                 [ 36%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_exceptions.py::test_unhandled_info_iwarn PASSED                                                [ 37%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb03rd PASSED                                                             [ 38%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb03vd_mb03vy_ex PASSED                                                   [ 39%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb03wd_ex PASSED                                                          [ 40%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb05md PASSED                                                             [ 41%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb05md_warning PASSED                                                     [ 42%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb::test_mb05nd PASSED                                                             [ 43%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb_docparse[mb03wd-SlycotResultWarning-erange0-checkvars0] PASSED                  [ 44%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb_docparse[mb05md-SlycotResultWarning-erange1-checkvars1] PASSED                  [ 45%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb_docparse[mb05nd-SlycotArithmeticError-erange2-checkvars2] PASSED                [ 46%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mb.py::test_mb_docparse[mc01td-SlycotResultWarning-erange3-checkvars3] PASSED                  [ 47%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mc.py::test_mc::test_mc01td PASSED                                                             [ 48%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mc.py::test_mc::test_mc01td_D PASSED                                                           [ 49%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_mc.py::test_mc::test_mc01td_warnings PASSED                                                    [ 50%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_sb.py::test_sb02mt PASSED                                                                      [ 50%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_sb.py::test_sb10ad PASSED                                                                      [ 51%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_sb.py::test_sb10jd PASSED                                                                      [ 52%]
Slycot/_skbuild/linux-x86_64-3.8/cmake-install/slycot/tests/test_sb.py::test_sb10fd %                                                                                  [ben@voyagerS9:~/src]%                                                                                                                                             [0] 

[repagh]

I just tried compiling on the latest master (8dc3069) and I still get a segmentation fault on a local compile. I am doing the same thing that was (and is) working on v0.3.5:

conda create -n test-slycot python=3.7 numpy
conda activate test-slycot 
conda install -c conda-forge gcc openblas scikit-build cmake
python setup.py install

I followed these instructions, and can confirm that I now get the very same result; a segfault on slycot load.

Libraries linked in without indicating blas vendor:

(test-slycot) repa$ otool -L ~/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot-0.3.5.167-py3.7-macosx-10.14-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so 
/Users/repa/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot-0.3.5.167-py3.7-macosx-10.14-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so:
    slycot/_wrapper.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libopenblas.0.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
    @rpath/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
    /Users/repa/anaconda3/envs/test-slycot/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

and same result with -DBLA_VENDOR=Apple

repa$ otool -L ~/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot-0.3.5.167-py3.7-macosx-10.14-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so 
/Users/repa/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot-0.3.5.167-py3.7-macosx-10.14-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so:
    slycot/_wrapper.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    @rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
    @rpath/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
    /Users/repa/anaconda3/envs/test-slycot/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

The gfortran in this case is from conda-forge.

For posterity, it would be good if @repagh or others who have this working on MacOS can say something about their development environment, in particular where gfortran and openblas are coming from.

If I create a conda package, I get the following link information:

repa$ otool -L ~/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot/_wrapper.cpython-37m-darwin.so 
/Users/repa/anaconda3/envs/test-slycot/lib/python3.7/site-packages/slycot/_wrapper.cpython-37m-darwin.so:
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    @rpath/libgfortran.4.dylib (compatibility version 5.0.0, current version 5.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

This one works, it only trips over the fresh XERBLA test, it seems to insist on picking up the Accelerate XERBLA, which exits the process with an exit code 1. So much for LAPACK standardization.

Note that it significantly picks up a newer Fortran library. It also uses clang.

    clang:                9.0.1-default_hf57f61e_0   conda-forge
    clang-tools:          9.0.1-default_hf57f61e_0   conda-forge
    clang_osx-64:         9.0.1-h05bbb7f_0           conda-forge
    clangxx:              9.0.1-default_hf57f61e_0   conda-forge
    cmake:                3.17.0-hd28f656_0          conda-forge
    compiler-rt:          9.0.1-h6a512c6_3           conda-forge
    compiler-rt_osx-64:   9.0.1-h99342c6_3           conda-forge
    expat:                2.2.9-h4a8c4bd_2           conda-forge
    gfortran_impl_osx-64: 7.3.0-hf4212f2_2           conda-forge
    gfortran_osx-64:      7.3.0-h22b1bf0_11          conda-forge

I don't have a definitive reason yet. I might have had the clang + newer Fortran in the previous environment I tested in.

Update

I was able to successfully build and test with a conda environment, and the fortran from conda-forge. This picked up the standard clang from OSX.

conda create -n slycot-test2 python=3.7 numpy
conda activate slycot-test2
conda install -c conda-forge scikit-build gfortran_osx-64 pytest scipy
python setup.py install -- -DBLA_VENDOR=Apple

[bnavigator]

This one works, it only trips over the fresh XERBLA test, it seems to insist on picking up the Accelerate XERBLA, which exits the process with an exit code 1. So much for LAPACK standardization.

Do you mean test_xerbla_override fails because it prints more than it should or does test_sb10fd exit as in https://github.com/python-control/Slycot/issues/126#issuecomment-630513188? Or both?

The MacOS X Accelerate builds on Travis do not exit with exit code 1. Neither before nor after the merge of #128.

[repagh]

Do you mean test_xerbla_override fails because it prints more than it should or does test_sb10fd exit as in #126 (comment)? Or both?

Upon running pytest --pyargs slycot:

This is the end of the blurp:

E               subprocess.CalledProcessError: Command '['/Users/repa/anaconda3/envs/test-slycot/bin/python', '-c', '\nfrom slycot._wrapper import ab08nd\n# equil=\'X\' is invalid\nout = ab08nd(1, 1, 1, [1], [1], [1], [1], equil=\'X\')\nprint("INFO={}".format(out[-1]))\n']' returned non-zero exit status 1.

../../../anaconda3/envs/test-slycot/lib/python3.7/subprocess.py:512: CalledProcessError

So it exits with a '1' exit code.

The curious fact is that the test within the conda build does not give that error.

[bnavigator]

That's most likely an ImportError before even calling ab08nd.

Could you try #130, please?

The import jungle with pytest, --pyargs pytest, sys.path, cwd, PYTHONPATH, ... etc is a mess. I am not completely innocent in that regard.

[repagh]

That's most likely an ImportError before even calling ab08nd.

Could you try #130, please?

You are right.ImportError, because the xerbla test apparently runs in the current folder. #130 works fine.

[roryyorke]

I couldn't do the release last Sunday, sorry -- some work came up.

Are we ready to release?

[bnavigator]

The MacOS segfault at import errors still pose a problem now that not only @murrayrm but also @repagh gets it after python setup.py install in a conda environment.

[roryyorke]

@repagh added an update to https://github.com/python-control/Slycot/issues/98#issuecomment-630794193 saying he could build, if I read correctly. I'll wait for him to respond.

[murrayrm]

I tried the following:

conda create -n slycot-test2 python=3.7 numpy
conda activate slycot-test2
conda install -c conda-forge scikit-build gfortran_osx-64 pytest scipy
python setup.py install -- -DBLA_VENDOR=Apple

When I run python setup.py test, python setup.py install -- -DBLA_VENDOR=Apple or just python -c "import slycot" I still get a segfault.

Output from otools:

/Users/murray/anaconda3/envs/slycot-test2/lib/python3.7/site-packages/slycot-0.3.5.175-py3.7-macosx-10.15-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so:
    _wrapper.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    @rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
    @rpath/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
    /Users/murray/anaconda3/envs/python3.7-slycot/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

I suspect the problem is that I am not getting a compatible version of compiler and math libraries? Here is the output of conda list:

# packages in environment at /Users/murray/anaconda3/envs/slycot-test2:
#
# Name                    Version                   Build  Channel
attrs                     19.3.0                     py_0    conda-forge
blas                      1.0                         mkl  
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
cmake                     3.14.0               haff7e42_0  
distro                    1.4.0                      py_0    conda-forge
expat                     2.2.9                h4a8c4bd_2    conda-forge
gfortran_osx-64           4.8.5                h22b1bf0_8    conda-forge
importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
intel-openmp              2019.4                      233  
krb5                      1.17.1               hddcf347_0  
libblas                   3.8.0                    14_mkl    conda-forge
libcblas                  3.8.0                    14_mkl    conda-forge
libcurl                   7.69.1               hc0b9707_0    conda-forge
libcxx                    10.0.0                        1  
libedit                   3.1.20181209         hb402a30_0  
libffi                    3.3                  h0a44026_1  
libgfortran               3.0.1                h93005f0_2  
liblapack                 3.8.0                    14_mkl    conda-forge
libssh2                   1.9.0                h39bdce6_2    conda-forge
mkl                       2019.4                      233  
mkl-service               2.3.0            py37hfbe908c_0  
mkl_fft                   1.0.15           py37h5e564d8_0  
mkl_random                1.1.0            py37ha771720_0  
more-itertools            8.3.0                      py_0    conda-forge
ncurses                   6.2                  h0a44026_1  
ninja                     1.10.0               ha1b3eb9_0    conda-forge
numpy                     1.18.1           py37h7241aed_0  
numpy-base                1.18.1           py37h3304bdc_1  
openssl                   1.1.1g               h0b31af3_0    conda-forge
packaging                 20.4               pyh9f0ad1d_0    conda-forge
pip                       20.0.2                   py37_3  
pluggy                    0.13.1           py37hc8dfbb8_1    conda-forge
py                        1.8.1                      py_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pytest                    5.4.2            py37hc8dfbb8_0    conda-forge
python                    3.7.7                hf48f09d_4  
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  h1de35cc_0  
rhash                     1.3.8                ha12b0ac_0  
scikit-build              0.11.1           py37h570ac47_0    conda-forge
scipy                     1.2.1            py37hbd7caa9_1    conda-forge
setuptools                46.4.0                   py37_0  
six                       1.14.0                   py37_0  
slycot                    0.3.5.175                 dev_0    <develop>
sqlite                    3.31.1               h5c1f38d_1  
tk                        8.6.8                ha441bb4_0  
wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
wheel                     0.34.2                   py37_0  
xz                        5.2.5                h1de35cc_0  
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11               h1de35cc_3  

As before, I don't think this is a blocker for releasing v0.4.0 and worst case I can always do a conda install of the conda-forge version. It's still odd, though.

[moorepants]

One thing is that you are installing a mix of conda packages from defaults and conda forge. I think its best to configure conda to install everything from conda forge. See this: https://conda-forge.org/docs/user/tipsandtricks.html#how-to-fix-it and https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html.

[moorepants]

Maybe using conda create -c conda-forge -n slycot-test2 python=3.7 numpy would do it.

[murrayrm]

Loading everything from condo-forge seems to work. I now get things to compile (!).

Running python setup.py test gives 4 errors. The last of them (pretty much same as the rest) says

ERROR: slycot.tests.test_mc (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: slycot.tests.test_mc
Traceback (most recent call last):
  File "/Users/murray/anaconda3/envs/slycot-test2/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/Users/murray/anaconda3/envs/slycot-test2/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/Users/murray/Dropbox/macosx/src/slycot/murrayrm/slycot/tests/test_mc.py", line 8, in <module>
    from slycot import mc01td
ImportError: cannot import name 'mc01td' from 'slycot' (/Users/murray/Dropbox/macosx/src/slycot/murrayrm/slycot/__init__.py)

Running pytest --pyargs slycot gives 1 failure:

_________________________ test_example[ab13fd_example] _________________________

examplefun = <function ab13fd_example at 0x116ab9ef0>
capsys = <_pytest.capture.CaptureFixture object at 0x116b951d0>
recwarn = WarningsRecorder(record=True)

    @pytest.mark.parametrize('examplefun', examplefunctions)
    def test_example(examplefun, capsys, recwarn):
        """
        Test the examples.

        Test that all the examples work, produce some (unchecked) output but no
        exceptions or warnings.
        """
        examplefun()
        captured = capsys.readouterr()
        assert len(captured.out) > 0
        assert not captured.err
>       assert not recwarn
E       assert not WarningsRecorder(record=True)

/Users/murray/anaconda3/envs/slycot-test2/lib/python3.7/site-packages/slycot-0.3.5.175-py3.7-macosx-10.15-x86_64.egg/slycot/tests/test_examples.py:29: AssertionError

[roryyorke]

This diff turns warnings into errors, and lets one see the warning (see pytest docs)

diff --git a/slycot/tests/test_examples.py b/slycot/tests/test_examples.py
index b926e2e..bb9cdae 100644
--- a/slycot/tests/test_examples.py
+++ b/slycot/tests/test_examples.py
@@ -13,9 +13,10 @@ from slycot import examples
 examplefunctions = [fun for (fname, fun) in getmembers(examples)
                     if isfunction(fun) and "_example" in fname]

+pytestmark = pytest.mark.filterwarnings("error")

 @pytest.mark.parametrize('examplefun', examplefunctions)
-def test_example(examplefun, capsys, recwarn):
+def test_example(examplefun, capsys):
     """
     Test the examples.

@@ -26,4 +27,3 @@ def test_example(examplefun, capsys, recwarn):
     captured = capsys.readouterr()
     assert len(captured.out) > 0
     assert not captured.err
-    assert not recwarn

I tested this in an editable install (pip install -e .), and noticed two things of interest:

1. pytest --pyargs slycot.tests.test_examples would run the whole suite; I couldn't figure out why until I remembered the addopts = --pyargs slycot line in setup.cfg
2. When I removed that line, and tried pytest -Werror --pyargs slycot.tests.test_examples, I got this:

slycot/tests/test_examples.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
slycot/examples.py:52: in ab08nd_example
    from scipy.linalg import eigvals
../venv-dev-slycot/lib/python3.6/site-packages/scipy/linalg/__init__.py:196: in <module>
    from .basic import *
../venv-dev-slycot/lib/python3.6/site-packages/scipy/linalg/basic.py:17: in <module>
    from ._solve_toeplitz import levinson
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

__init__.pxd:918: RuntimeWarning
=============================================== short test summary info ===============================================
FAILED slycot/tests/test_examples.py::test_example[ab08nd_example] - RuntimeWarning: numpy.ufunc size changed, may i...
============================================ 1 failed, 12 passed in 0.34s =============================================

So, if I read that right, it's not only on macOS that Slycot gets its numpy libraries mixed up.

This doesn't happen with pytest --pyargs slycot.tests or pytest --pyargs slycot, which I don't understand.

Finally, if I run pytest -Werror --pyargs slycot, I get:

>               warn(globals()[warning](fmessage, iwarn, info))
E               slycot.exceptions.SlycotResultWarning: 
E               Entry ``P(x)`` is the zero polynomial.

slycot/exceptions.py:241: SlycotResultWarning
=============================================== short test summary info ===============================================
FAILED slycot/tests/test_mc.py::test_mc::test_mc01td_warnings - slycot.exceptions.SlycotResultWarning: 
============================================ 1 failed, 107 passed in 3.73s ============================================

[roryyorke]

That ufunc size warning is triggered by the from scipy.linalg import eigvals in ab08nd_example. General advice seems to be to ignore it; see [1], [2].

In our case, we can work around it by importing scipy.linalg at module level in test_examples.py (I don't know why this works; perhaps it is pytest resetting import filters, as suggest in a comment in [2].)

[1] https://github.com/numpy/numpy/pull/432/commits/170ed4e33d6196d724dc18ddcd42311c291b4587 [2] https://github.com/numpy/numpy/issues/14920

[repagh]

I tried the following:

conda create -n slycot-test2 python=3.7 numpy
conda activate slycot-test2
conda install -c conda-forge scikit-build gfortran_osx-64 pytest scipy
python setup.py install -- -DBLA_VENDOR=Apple

When I run python setup.py test, python setup.py install -- -DBLA_VENDOR=Apple or just python -c "import slycot" I still get a segfault.

Output from otools:

/Users/murray/anaconda3/envs/slycot-test2/lib/python3.7/site-packages/slycot-0.3.5.175-py3.7-macosx-10.15-x86_64.egg/slycot/_wrapper.cpython-37m-darwin.so:
  _wrapper.cpython-37m-darwin.so (compatibility version 0.0.0, current version 0.0.0)
  /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
  @rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
  @rpath/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
  /Users/murray/anaconda3/envs/python3.7-slycot/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
  @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

That is just maddening.

I got gfortran_osx-64 version 7.3.0 installed with those same instructions. That links to @rpath/libgfortran4.dylib, and does not link to libquadmath, and everything then simply works.

[bnavigator]

Running python setup.py test gives 4 errors.

Deprecated command, wich does not use pytest. Error looks like an old slycot ~wrapper~ without mc01td got imported from a src dir outside of the venv.

[bnavigator]

Running pytest --pyargs slycot gives 1 failure:

@murrayrm, could you please check with #133, which failure in the examples that would be? ab13fd_example is different from @roryyorke's failure in ab08nd_example

[murrayrm]

Ran #133 and got the following output from pytest --pyargs slycot

_________________________ test_example[ab13fd_example] _________________________

examplefun = <function ab13fd_example at 0x118cf8290>
capsys = <_pytest.capture.CaptureFixture object at 0x118e1e990>
recwarn = WarningsRecorder(record=True)

    @pytest.mark.parametrize('examplefun', examplefunctions)
    @pytest.mark.filterwarnings("ignore:numpy.dtype size changed")
    @pytest.mark.filterwarnings("ignore:numpy.ufunc size changed")
    def test_example(examplefun, capsys, recwarn):
        """
        Test the examples.

        Test that all the examples work, produce some (unchecked) output but no
        exceptions or warnings.
        """
        examplefun()
        captured = capsys.readouterr()

        # fail for first in order
        failconditions = [
            ((not len(captured.out) > 0), "Example {} did not print any results\n"),
            (captured.err, "Example {} wrote to stderr\n"),
            (recwarn,  "Example {} produced a warning.\n")]
        for failed, msgfmt  in failconditions:
            if failed:
                pytest.fail(msgfmt.format(examplefun.__name__) +
                            "Captured output:\n{}\n"
                            "Captured stderr:\n{}\n"
                            "Captured warnings:\n{}\n"
                            "".format(captured.out,
                                      captured.err,
>                                     [w.message for w in recwarn]))
E               Failed: Example ab13fd_example produced a warning.
E               Captured output:
E               The stability radius is
E               0.003919647231698771
E               The minimizing omega is
E               0.9896652038894939
E               
E               Captured stderr:
E               
E               Captured warnings:
E               [SlycotResultWarning('\nFailed to compute beta(A) within the specified tolerance.\nNevertheless, the returned value is an upper bound on beta(A);')]

../anaconda3/envs/slycot-test2/lib/python3.7/site-packages/slycot-0.3.5.179-py3.7-macosx-10.15-x86_64.egg/slycot/tests/test_examples.py:44: Failed

[bnavigator]

Interesting. Should we increase the default tolerance in the example or introduce an option to ignore specific warnings in examples?

[bnavigator]

E               Failed: Example ab13fd_example produced a warning.
E               Captured output:
E               The stability radius is
E               0.003919647231698771
E               The minimizing omega is
E               0.9896652038894939
E               
E               Captured stderr:
E               
E               Captured warnings:
E               [SlycotResultWarning('\nFailed to compute beta(A) within the specified tolerance.\nNevertheless, the returned value is an upper bound on beta(A);')]

Allowing it in af98e9fca2d94e2ade6518034eea3405ee2fa0bd

[roryyorke]

I've at last released v0.4.0.

@moorepants are you (still?) a maintainer for slycot on PyPI? If so, could you please upload it?

@murrayrm I've at last created a PyPI account, my username is roryyorke.

[roryyorke]

• [ ] upload to PyPI
• [ ] file PR with conda-forge feedstock

[moorepants]

It looks like I do have permission to upload. I assume you want me to:

git checkout v0.4.0
python setup.py sdist
twine upload dist/slycot-0.4.0.0.tar.gz

[moorepants]

Note too that the conda forge pot will create a PR once PyPi is updated.

[murrayrm]

@roryyorke: Added you as a maintainer on PyPi.