Include osx_arm64 builds

[sanurielf]

Checklist

• [x] Used a personal fork of the feedstock to propose changes
• [x] Bumped the build number (if the version is unchanged)
• [x] Reset the build number to 0 (if the version changed)
• [x] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• [x] Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[sanurielf]

Hi reviewers/mainteners,

I don't know pyomo internals so it will be great if you can check why the OSX arm64 is falling.

I have tested locally in a M1 and it worked.

[ngam]

Please add python, cython, and cross_python to requirements/build like below and then copy this: @conda-forge-admin, please rerender

requirements:
  build:
    - {{ compiler('c') }}
    - {{ compiler('cxx') }}
+   - python                                 # [build_platform != target_platform]
+   - cython                                 # [build_platform != target_platform]
+   - cross-python_{{ target_platform }}     # [build_platform != target_platform]

[github-actions[bot]]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/pyomo-feedstock/actions/runs/1685563623.

[ngam]

Currently the CI doesn't have any arm64 machines, so everything gets cross-compiled. Hence you need cython, python, and cross_python in the build requirements. If you do that, you will get something like this. Note: "build_platform": "osx-64" and "target_platform": "osx-arm64".

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "pyomo-6.2-py39hc057bc8_0": {
    "recipe": {
      "CONDA_BUILD_SYSROOT": "/opt/MacOSX11.0.sdk",
      "build_platform": "osx-64",
      "c_compiler": "clang",
      "c_compiler_version": "11",
      "channel_targets": "conda-forge main",
      "cxx_compiler": "clangxx",
      "cxx_compiler_version": "11",
      "python": "3.9.* *_cpython",
      "target_platform": "osx-arm64"
    }
  }
}

Also, never mind... I thought this could be a noarch candidate, but it actually cannot be. There's a lot of c stuff going behind the scene I guess.

[sanurielf]

Thanks a lot @ngam . Now builds are passing.

@reviewers if you consider this is ok, please merge it whenever possible.

[mrmundt]

@sanurielf - We are looking into this.

As a general rule, we do not release builds for architectures against which we are not currently testing, and we have no testing for Mac OSX ARM64 processors at this time. I'll keep you posted.

[sanurielf]

@mrmundt thanks for letting me know.

I wasn't aware we don't have tests against the OSX arm64 environment. I'm currently using it from OSX_arm64 (built from source) and seems to work fine.

I know that further analysis must be done before releasing it, so let me try to take a look at the GitHub actions you have for testing on main repo and pull-request some OSX arm64 builds.

[ngam]

@mrmundt + @blnicho: perhaps you should add a few tests to be run on this CI here. I can help you with that. I know you do the imports (which is a good start) but you have two options depending on what you do upstream:

1. If your test suite can pass within an hour or so, then you can just run it entirely here
2. If your test suite is super expansive (takes many hours, etc.), you could run a few basic examples (the tutorial if you have one) and a few basic tests.

Generally speaking, as @sanurielf indicates above, cross-compiling shouldn't break much, but you know your package more than we do, so it is your call. Btw, adding linux_aarch64 and linux_ppc64le is also rather straightforward (I can initiate that for you in another PR). Anyway, I would be quite surprised if anything breaks by supporting the alternative arches. It's prudent to test, nonetheless, and I highly recommend you do proper testing here if you are serious about support as you say you are (the conda CIs use pretty older OSes and compilers in order to support basically as many machines in circulation as possible, so something would have already broken if you are worried about arm-osx64)

[ngam]

It seems to me from https://github.com/Pyomo/pyomo/runs/4807350499?check_suite_focus=true (under Run Pyomo tests) that your tests take on the order of 10 mins. If so, you should just run them all here.

However, just to be clear: The tests are usually not run when you cross-compile. Hence, test_on_native_only: true in conda-forge.yaml as edited above by @sanurielf.

  provider:
    win: azure
  conda_forge_output_validation: true
+ build_platform:
+   osx_arm64: osx_64
+ test_on_native_only: true

and we have no testing for Mac OSX ARM64 processors at this time

Yep, difficult to get public CI for osx-arm64 support. Best you could probably do is to run on someone's machine. I can run your entire test suite on osx-arm64 and upload the results for you --- using this conda recipe.

[ngam]

Also, I see you have a specific mpi run/test going on. Not sure what's that all about, but it is possible (and relatively straightforward) to support different mpi stuff here and you will end up with different builds depending the mpi support. For more: https://conda-forge.org/docs/maintainer/knowledge_base.html#message-passing-interface-mpi

[sjpfenninger]

Is there any timeline on when this might get merged? In several projects, Pyomo is our only remaining dependency without an osx_arm64 build.

[mrmundt]

Hello, @sjpfenninger - thank you for pinging us about this. It fell off our radar.

@ngam , we do not have an arm64 MacOS machine (though we do have plans to eventually purchase one once appropriate funding comes through). We would appreciate any assistance you can provide here - you mentioned being able to maybe build some tests into this CI?

[ngam]

@conda-forge-admin, please rerender

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict. Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

[ngam]

@mrmundt, there is no osx-arm64 on public CIs yet. Hence, any testing is moot. However, what we can do here is to test it locally.

[ngam]

All of conda-forge M1 releases are experimental with no testing. However, the users will file issues if they discover issues. Generally, if things pass on osx-64, they will likely be fine on osx-arm64. Otherwise, you will see build issues.

I started another PR since this one is now outdated. #66. I will try to download the artifacts and run some tests locally for you to see what's up...

[ngam]

Hello, @sjpfenninger - thank you for pinging us about this. It fell off our radar.

@ngam , we do not have an arm64 MacOS machine (though we do have plans to eventually purchase one once appropriate funding comes through). We would appreciate any assistance you can provide here - you mentioned being able to maybe build some tests into this CI?

I wouldn't worry about buying an M1 machine just for this. As I said, pyomo users with M1 machines (which likely will be many) will test and report back. Just put a disclaimer that the M1 release is experimental (you can do this like JAX, e.g. https://github.com/google/jax/blob/2153a57e5c880690bc75f509f788dbc71a8cd618/jax/_src/lib/__init__.py#L31-L36) or in your readme or something. For production uses, most likely, users will be using linux-64 anyway.

[ngam]

Here you go, I ran some of your tests in this feedstock:

``` Last login: Tue May 31 23:02:20 on ttys000 ~$ mamba create -n pyomo_test -c ngam -c conda-forge pyomo __ __ __ __ / \ / \ / \ / \ / \/ \/ \/ \ ███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ \____ / / \_/ \_/ \_/ \ o \__, / _/ \_____/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝ mamba (0.23.3) supported by @QuantStack GitHub: https://github.com/mamba-org/mamba Twitter: https://twitter.com/QuantStack █████████████████████████████████████████████████████████████ Looking for: ['pyomo'] ngam/osx-arm64 884.0 B @ 2.6kB/s 0.3s ngam/noarch 125.0 B @ 364.0 B/s 0.3s conda-forge/osx-arm64 2.5MB @ 3.2MB/s 0.9s conda-forge/noarch 8.3MB @ 3.7MB/s 2.4s Transaction Prefix: /Users/ngam/.Mambaforge-MacOSX-arm64/envs/pyomo_test Updating specs: - pyomo Package Version Build Channel Size ──────────────────────────────────────────────────────────────────────────────────────── Install: ──────────────────────────────────────────────────────────────────────────────────────── + bzip2 1.0.8 h3422bc3_4 conda-forge/osx-arm64 Cached + ca-certificates 2022.5.18.1 h4653dfc_0 conda-forge/osx-arm64 Cached + libcxx 14.0.4 h6a5c8ee_0 conda-forge/osx-arm64 Cached + libffi 3.4.2 h3422bc3_5 conda-forge/osx-arm64 Cached + libzlib 1.2.12 h90dfc92_0 conda-forge/osx-arm64 Cached + ncurses 6.3 h07bb92c_1 conda-forge/osx-arm64 Cached + openssl 3.0.3 ha287fd2_0 conda-forge/osx-arm64 Cached + pip 22.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached + ply 3.11 py_1 conda-forge/noarch 45kB + pyomo 6.4.1 py310hb07a4bc_1 ngam/osx-arm64 6MB + python 3.10.4 h14b404e_0_cpython conda-forge/osx-arm64 Cached + python_abi 3.10 2_cp310 conda-forge/osx-arm64 Cached + readline 8.1 hedafd6a_0 conda-forge/osx-arm64 Cached + setuptools 62.3.2 py310hbe9552e_0 conda-forge/osx-arm64 Cached + sqlite 3.38.5 h40dfcc0_0 conda-forge/osx-arm64 Cached + tk 8.6.12 he1e0b03_0 conda-forge/osx-arm64 Cached + tzdata 2022a h191b570_0 conda-forge/noarch Cached + wheel 0.37.1 pyhd8ed1ab_0 conda-forge/noarch Cached + xz 5.2.5 h642e427_1 conda-forge/osx-arm64 Cached + zlib 1.2.12 h90dfc92_0 conda-forge/osx-arm64 Cached Summary: Install: 20 packages Total download: 6MB ──────────────────────────────────────────────────────────────────────────────────────── Confirm changes: [Y/n] y ply 44.8kB @ 224.8kB/s 0.2s pyomo 5.6MB @ 1.9MB/s 2.9s Preparing transaction: done Verifying transaction: done Executing transaction: done To activate this environment, use $ mamba activate pyomo_test To deactivate an active environment, use $ mamba deactivate ~$ mamba activate pyomo_test (pyomo_test) ~$ pyomo --help usage: pyomo [-h] [--version] {build-extensions,check,convert,download-extensions,help,install-extras,model-viewer,run,solve,test-solvers} ... This is the main driver for the Pyomo optimization software. options: -h, --help show this help message and exit --version show program's version number and exit subcommands: {build-extensions,check,convert,download-extensions,help,install-extras,model-viewer,run,solve,test-solvers} build-extensions Build compiled extension modules check Check a model for errors. convert Convert a Pyomo model to another format download-extensions Download compiled extension modules help Print help information. install-extras Install "extra" packages that Pyomo can leverage. model-viewer Run the Pyomo model viewer run Execute a command from the Pyomo bin (or Scripts) directory. solve Optimize a model test-solvers Test Pyomo solvers ------------------------------------------------------------------------- Pyomo supports a variety of modeling and optimization capabilities, which are executed either as subcommands of 'pyomo' or as separate commands. Use the 'help' subcommand to get information about the capabilities installed with Pyomo. Additionally, each subcommand supports independent command-line options. Use the -h option to print details for a subcommand. For example, type pyomo solve -h to print information about the `solve` subcommand. (pyomo_test) ~$ python3 Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:37) [Clang 12.0.1 ] on darwin Type "help", "copyright", "credits" or "license" for more information. i>>> import pyomo >>> import pyomo.checker >>> import pyomo.checker.plugins >>> import pyomo.common >>> import pyomo.contrib >>> import pyomo.core >>> import pyomo.gdp >>> import pyomo.gdp.plugins >>> import pyomo.opt >>> import pyomo.repn >>> import pyomo.util >>> import pyomo.solvers >>> >>> pyomo.__version__ '6.4.1' >>> import platform >>> platform.platform() 'macOS-12.4-arm64-arm-64bit' >>> ```

[ngam]

And I ran your base unit tests too (from the pyomo repo):

(one docs error, which I won't investigate or bother with)

``` (pyomo_test) ~/Repos/pyomo$ python3 -m pytest . ================================== test session starts =================================== platform darwin -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 rootdir: /Users/ngam/Repos/pyomo, configfile: setup.cfg collected 13048 items / 1 error / 30 skipped ========================================= ERRORS ========================================= _________________ ERROR collecting doc/OnlineDocs/tests/test_examples.py _________________ doc/OnlineDocs/tests/test_examples.py:198: in Test.add_baseline_test( E AttributeError: type object 'Test_expr' has no attribute 'add_baseline_test' ================================ short test summary info ================================= ERROR doc/OnlineDocs/tests/test_examples.py - AttributeError: type object 'Test_expr' h... !!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!! ============================== 30 skipped, 1 error in 1.87s ============================== (pyomo_test) ~/Repos/pyomo$ ```

[ngam]

Anyway, I would say just merge the other PR, #66, and add a disclaimer if you wish. Best of luck :)

btw, sorry about the late response. I don't get notifications, but I passed by here accidentally again. In real life, I happened to know some of the people involved in this project (from the CMU days). Hopefully, next time I will see your tag earlier

[sanurielf]

Hi @ngam

Please keep this PR unique for the OSX arm integration. There is a way to provide CI on arm64 builds as mentioned in this PR.

I've tested it using PYOMO github actions and my MBP and the first test results are here:

https://github.com/sanurielf/pyomo/runs/6694616199?check_suite_focus=true

As you can see there are more dependencies to have on osx ARM64 before releasing PYOMO. I will go over each one to get a possible solution.

@mrmundt is the casadi package mandatory for pyomo?

https://github.com/sanurielf/pyomo/runs/6694616199?check_suite_focus=true#step:13:109

[mrmundt]

@sanurielf - casadi is NOT a requirement. It's an optional package that is necessary for Pyomo DAE but not core capability. The only requirement for core capability is ply.

[ngam]

Alright, closing #66. Best of luck.