Closed hblumberg closed 1 year ago
Thanks for reporting, @hblumberg. Not sure that I have more guidance, but perhaps @willirath has experience with osx-arm64
?
Now that we have a no-arch
version on conda since https://github.com/conda-forge/parcels-feedstock/pull/98, my experience is that macOS works easier. So closing this issue (for now)
This issue remains in the development build. environment_py3_osx.yml
asks for clang_osx_64
which cause the build to fail on my m2 macbook, unless I run everything under Rosetta. I have fixed it by changing it to clang_osx-arm64
in a new .yml file environment_py3_osx_arm64.yml
. Installing parcels under an environment created by this .yml file works for me now.
I am not going to make this part of my existing patch request (userPartitionMPI
), because it is an entirely different issue. Attached is my new build file.
name: parcels
channels:
- conda-forge
dependencies:
- python>=3.8
- cgen
- clang_osx-arm64
- ffmpeg>=3.2.3
- git
- mpi4py>=3.0.1
- mpich>=3.2.1
- jupyter
- matplotlib-base>=2.0.2
- netcdf4>=1.1.9
- numpy>=1.9.1
- platformdirs
- psutil
- py>=1.4.27
- pymbolic
- scipy>=0.16.0
- tqdm
- trajan
- xarray>=0.10.8
- dask>=2.0
- cftime>=1.3.1
- nbval
- scikit-learn
- pykdtree
- zarr>=2.11.0
# Formatting
- black
- isort
# Testing
- pytest
- pytest-html
- coverage
# Linting
- flake8>=2.1.0
- pre_commit
- pydocstyle
# Docs
- ipython
- numpydoc
- nbsphinx
- sphinx<6
- pandoc>1.12.1,<3
- pydata-sphinx-theme
- sphinx-autobuild
- myst-parser
Thanks @JamiePringle, for providing this patch. But rather than adding another environment*.yml
file, I wonder if we could not go down the route of https://github.com/conda-forge/parcels-feedstock/pull/98 and provide only one environment file for all architectures.
A bit like how it's done in https://github.com/conda-forge/parcels-feedstock/blob/50ab3a48c31416a926185101cfbb339de49940d5/recipe/meta.yaml#L21-L48
Does @willirath, @ocefpaf or @ngam perhaps know how to do that for the environment*.yml
files in https://github.com/OceanParcels/parcels that conda uses, too?
These two lines,
ensures that compilers will be available for Windows and "not windows." However, I'm not sure the [unix]
pre-processor selector covers osx-arm64
, probably not b/c that arch is still kind of new. I'll ask this in the conda forums.
With that said we can:
osx-arm64
(probably would require some work upstream to recognize that as OS-noarch)noarch
and an arch'ed parcels package that adds the compilers and depends on the core one.IMO 3 is the best solution b/c it doesn't depend on any action from upstream.
If I am reading this correctly, the error seems to be that the env yml upstream is asking for a specific compiler. The user shouldn’t need to ask for the compilers at all. Can you please by removing the compiler in the env yml file? If you want a generic compiler for osx (both arches) it is “c-compiler” for the default one
@JamiePringle’s solution is good, but I would go even farther and remove the compiler (clang) entry completely from that file
If that doesn’t work, we will follow @ocefpaf’s recommendation. I believe unix works for osx-arm64 already, btw, but we could double check
I've created #1416 to test if we can move to an environment file that doesn't include compilers. If this passes on the Github Action CI, then we should try to implement this?
I've created https://github.com/OceanParcels/parcels/pull/1416 to test if we can move to an environment file that doesn't include compilers. If this passes on the Github Action CI, then we should try to implement this?
OK, so removing the compilers from each of the three environment files doesn't break the CI (don't mind the red cross in https://github.com/OceanParcels/parcels/pull/1416; that's because of a strange bug in ReadTheDocs, see #1417). So that seems that we can even have an option 4 on @ocefpaf's list above; or am I missing something?
Could your tests be picking up any compilers available in the CIs? If so, that won't help end users, unless you amend the docs and tell them to install compilers from the system or with conda.
Could your tests be picking up any compilers available in the CIs? If so, that won't help end users, unless you amend the docs and tell them to install compilers from the system or with conda.
Well, this is for the Developers version of the environments, so we can expect them to have a compiler available (and will explain that in the documentation at https://docs.oceanparcels.org/en/latest/installation.html#installation-for-developers
'Normal' users would use the condo package at https://github.com/conda-forge/parcels-feedstock, so would not deal with these environment files anyways
Hmm, adding [unix]
behind the mpi4py
and mpich
packages (because they don't exist for windows doesn't seem to work in https://github.com/OceanParcels/parcels/pull/1416/commits/079777c0480d479010382ec96e542f0b31b7821d.
- mpi4py>=3.0.1 # [unix]
- mpich>=3.2.1 # [unix]
This gives an error in Windows CI that mpich does not exist
Anyone has an idea how to specify that certain packages should only be installed under unix architectures in a conda environment.yml file?
Conda doesn't support that. It is a long standing issue. See https://github.com/conda/conda/issues/8089
You can do it with mamba/micromamba though. See https://github.com/mamba-org/mamba/blob/20b477f4595e7d932034945fd122a5a40cd25c9c/libmamba/tests/data/env_file/env_2.yaml#L6 for an example.
OK, thanks for this information, @ocefpaf.
And would you have any advice on whether it's worth it to change to mamba/micromamba for the developer installation then? I don't have much experience with it, but from what I read it seems a newer/better implementation of conda?
I highly recommend micromamba. Now, with latest conda adopting the libmamba solver as its default solver, the solve times for all 3 are virtually the same and for most users those are equivalent However, micromamba is a smaller download, single binary (no Python base env to mess with), and faster overall for things beyond the solver.
OK, so how could we then change the https://github.com/OceanParcels/parcels/blob/noacrch-environment-file/environment.yml file so that it works with mamba?
And do we then also need to change the https://github.com/OceanParcels/parcels/blob/noacrch-environment-file/.github/workflows/integration-tests.yml and https://github.com/OceanParcels/parcels/blob/noacrch-environment-file/.github/workflows/unit-tests.yml files so that they use mamba and work with this new environment file?
Could you help us develop this in https://github.com/OceanParcels/parcels/pull/1416?
Could your tests be picking up any compilers available in the CIs? If so, that won't help end users, unless you amend the docs and tell them to install compilers from the system or with conda.
The package install should be picking up the correct compiler by simply listing parcels
(even on osx-arm64, see below).
``` ~$ micromamba create -n test_parcles parcels --dry-run conda-forge/osx-arm64 6.4MB @ 5.2MB/s 1.3s conda-forge/noarch 12.2MB @ 9.6MB/s 1.3s Transaction Prefix: /Users/ngam/.micromamba/envs/test_parcles Updating specs: - parcels Package Version Build Channel Size ────────────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────────────── + xz 5.2.6 h57fd34a_0 conda-forge Cached + libexpat 2.5.0 hb7217d7_1 conda-forge Cached + ncurses 6.4 h7ea286d_0 conda-forge Cached + libutf8proc 2.8.0 h1a8c8d9_0 conda-forge Cached + libjpeg-turbo 2.1.5.1 h1a8c8d9_0 conda-forge Cached + libwebp-base 1.3.1 hb547adb_0 conda-forge Cached + pthread-stubs 0.4 h27ca646_1001 conda-forge Cached + xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge Cached + python_abi 3.11 3_cp311 conda-forge Cached + libcxx 16.0.6 h4653b0c_0 conda-forge Cached + yaml 0.2.5 h3422bc3_2 conda-forge Cached + libffi 3.4.2 h3422bc3_5 conda-forge Cached + libzlib 1.2.13 h53f4e23_5 conda-forge Cached + libdeflate 1.18 h1a8c8d9_0 conda-forge Cached + ca-certificates 2023.7.22 hf0a4a13_0 conda-forge Cached + xorg-libxau 1.0.11 hb547adb_0 conda-forge Cached + llvm-openmp 16.0.6 h1c12783_0 conda-forge Cached + c-ares 1.19.1 hb547adb_0 conda-forge Cached + libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge Cached + aws-c-common 0.9.0 hb547adb_0 conda-forge Cached + libev 4.33 h642e427_1 conda-forge Cached + libiconv 1.17 he4db4b2_0 conda-forge Cached + icu 73.2 hc8870d7_0 conda-forge 12MB + bzip2 1.0.8 h3422bc3_4 conda-forge Cached + libedit 3.1.20191231 hc8eb9b7_2 conda-forge Cached + readline 8.2 h92ec313_1 conda-forge Cached + tapi 1100.0.11 he4954df_0 conda-forge Cached + libaec 1.0.6 hb7217d7_1 conda-forge Cached + lz4-c 1.9.4 hb7217d7_0 conda-forge Cached + lerc 4.0.0 h9a09cb3_0 conda-forge Cached + libcrc32c 1.1.2 hbdafb3b_0 conda-forge Cached + gflags 2.2.2 hc88da5d_1004 conda-forge Cached + snappy 1.1.10 h17c5cce_0 conda-forge Cached + re2 2023.03.02 hc5e2d97_0 conda-forge Cached + libabseil 20230125.3 cxx17_h13dd4ca_0 conda-forge Cached + hdf4 4.2.15 h8111dcc_6 conda-forge Cached + zstd 1.5.2 h4f39d0f_7 conda-forge Cached + libsqlite 3.43.0 hb31c410_0 conda-forge 834kB + libpng 1.6.39 h76d750c_0 conda-forge Cached + tk 8.6.12 he1e0b03_0 conda-forge Cached + openssl 3.1.2 h53f4e23_0 conda-forge Cached + libxcb 1.15 hf346824_0 conda-forge Cached + libgfortran5 12.3.0 ha3a6a3e_1 conda-forge Cached + libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge Cached + libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge Cached + aws-c-compression 0.2.17 he70778a_2 conda-forge Cached + aws-c-sdkutils 0.1.12 he70778a_1 conda-forge Cached + aws-checksums 0.1.17 he70778a_1 conda-forge Cached + libxml2 2.11.5 h25269f3_1 conda-forge 615kB + glog 0.6.0 h6da1cb0_0 conda-forge Cached + libprotobuf 4.23.3 hf32f9b9_0 conda-forge Cached + blosc 1.21.4 hc338f07_0 conda-forge Cached + libtiff 4.5.1 h23a1a89_1 conda-forge 365kB + freetype 2.12.1 hd633e50_1 conda-forge Cached + sigtool 0.1.3 h44b9a77_0 conda-forge Cached + libzip 1.10.1 ha0bc3c6_0 conda-forge 129kB + libevent 2.1.12 h2757513_1 conda-forge Cached + aws-c-cal 0.6.2 hb406d48_0 conda-forge 36kB + krb5 1.21.2 h92f50d5_0 conda-forge Cached + libssh2 1.11.0 h7a5bd25_0 conda-forge Cached + libnghttp2 1.52.0 hae82a92_0 conda-forge Cached + libgfortran 5.0.0 12_3_0_hd922786_1 conda-forge Cached + brotli-bin 1.0.9 h1a8c8d9_9 conda-forge Cached + libllvm15 15.0.7 h504e6bf_3 conda-forge Cached + libgrpc 1.56.2 h9075ed4_1 conda-forge Cached + orc 1.9.0 h858f345_1 conda-forge Cached + openjpeg 2.5.0 hbc2ba62_2 conda-forge Cached + lcms2 2.15 hd835a16_1 conda-forge Cached + libthrift 0.18.1 ha061701_2 conda-forge Cached + aws-c-io 0.13.32 h3c776e5_2 conda-forge 140kB + libcurl 8.2.1 hc52a3a8_0 conda-forge Cached + libopenblas 0.3.23 openmp_hc731615_0 conda-forge Cached + brotli 1.0.9 h1a8c8d9_9 conda-forge Cached + ld64_osx-arm64 609 hc4dc95b_14 conda-forge Cached + llvm-tools 15.0.7 h504e6bf_3 conda-forge Cached + libclang-cpp15 15.0.7 default_h5dc8d65_3 conda-forge Cached + aws-c-http 0.7.12 he297698_1 conda-forge 158kB + aws-c-event-stream 0.3.2 hcf14f3f_0 conda-forge 49kB + hdf5 1.14.1 nompi_h3aba7b3_100 conda-forge 3MB + libgoogle-cloud 2.12.0 h05652e3_1 conda-forge Cached + libblas 3.9.0 17_osxarm64_openblas conda-forge Cached + ld64 609 h89fa09d_14 conda-forge Cached + cctools_osx-arm64 973.0.1 h2a25c60_14 conda-forge Cached + clang-15 15.0.7 default_h5dc8d65_3 conda-forge Cached + aws-c-mqtt 0.9.5 h83b98fe_1 conda-forge 123kB + aws-c-auth 0.7.3 hab8b942_3 conda-forge 92kB + libcblas 3.9.0 17_osxarm64_openblas conda-forge Cached + liblapack 3.9.0 17_osxarm64_openblas conda-forge Cached + cctools 973.0.1 hd1ac623_14 conda-forge Cached + clang 15.0.7 hce30654_3 conda-forge Cached + aws-c-s3 0.3.14 h24e141d_3 conda-forge 78kB + clangxx 15.0.7 default_h610c423_3 conda-forge Cached + aws-crt-cpp 0.23.0 h04fc39a_5 conda-forge 224kB + aws-sdk-cpp 1.10.57 h0092a47_21 conda-forge 3MB + libnetcdf 4.9.2 nompi_h9fa6108_109 conda-forge 734kB + libarrow 13.0.0 h6e4acf5_0_cpu conda-forge 18MB + tzdata 2023c h71feb2d_0 conda-forge Cached + compiler-rt_osx-arm64 15.0.7 hf8d1dfb_1 conda-forge Cached + python 3.11.4 h47c9636_0_cpython conda-forge Cached + compiler-rt 15.0.7 hf8d1dfb_1 conda-forge Cached + clang_osx-arm64 15.0.7 h77e971b_3 conda-forge Cached + c-compiler 1.6.0 hd291e01_0 conda-forge Cached + wheel 0.41.2 pyhd8ed1ab_0 conda-forge 57kB + setuptools 68.1.2 pyhd8ed1ab_0 conda-forge Cached + pip 23.2.1 pyhd8ed1ab_0 conda-forge Cached + munkres 1.1.4 pyh9f0ad1d_0 conda-forge Cached + pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge Cached + cycler 0.11.0 pyhd8ed1ab_0 conda-forge Cached + zipp 3.16.2 pyhd8ed1ab_0 conda-forge Cached + tomli 2.0.1 pyhd8ed1ab_0 conda-forge Cached + iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge Cached + exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge Cached + pluggy 1.2.0 pyhd8ed1ab_0 conda-forge Cached + typing_extensions 4.7.1 pyha770c72_0 conda-forge Cached + appdirs 1.4.4 pyh9f0ad1d_0 conda-forge 13kB + idna 3.4 pyhd8ed1ab_0 conda-forge Cached + charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge Cached + certifi 2023.7.22 pyhd8ed1ab_0 conda-forge Cached + colorama 0.4.6 pyhd8ed1ab_0 conda-forge Cached + fasteners 0.17.3 pyhd8ed1ab_0 conda-forge Cached + pytz 2023.3 pyhd8ed1ab_0 conda-forge Cached + python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge Cached + six 1.16.0 pyh6c4a22f_0 conda-forge Cached + xyzservices 2023.7.0 pyhd8ed1ab_0 conda-forge Cached + fsspec 2023.6.0 pyh1a96a4e_0 conda-forge Cached + zict 3.0.0 pyhd8ed1ab_0 conda-forge Cached + pysocks 1.7.1 pyha2e5f31_6 conda-forge Cached + tblib 1.7.0 pyhd8ed1ab_0 conda-forge Cached + sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge Cached + packaging 23.1 pyhd8ed1ab_0 conda-forge Cached + cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge Cached + click 8.1.7 unix_pyh707e725_0 conda-forge Cached + asciitree 0.3.3 py_2 conda-forge Cached + toolz 0.12.0 pyhd8ed1ab_0 conda-forge Cached + locket 1.0.0 pyhd8ed1ab_0 conda-forge Cached + importlib-metadata 6.8.0 pyha770c72_0 conda-forge Cached + typing-extensions 4.7.1 hd8ed1ab_0 conda-forge Cached + tqdm 4.66.1 pyhd8ed1ab_0 conda-forge Cached + python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge Cached + pytest 7.4.0 pyhd8ed1ab_0 conda-forge Cached + partd 1.4.0 pyhd8ed1ab_0 conda-forge Cached + importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge Cached + platformdirs 3.10.0 pyhd8ed1ab_0 conda-forge Cached + pytools 2023.1.1 pyhd8ed1ab_0 conda-forge 71kB + kiwisolver 1.4.5 py311he4fd1f5_0 conda-forge 62kB + brotli-python 1.0.9 py311ha397e9f_9 conda-forge 325kB + markupsafe 2.1.3 py311heffc1b2_0 conda-forge Cached + pillow 10.0.0 py311h095fde6_0 conda-forge 46MB + numpy 1.25.2 py311hb8f3215_0 conda-forge Cached + tornado 6.3.3 py311heffc1b2_0 conda-forge 845kB + pyyaml 6.0.1 py311heffc1b2_0 conda-forge 189kB + psutil 5.9.5 py311he2be06e_0 conda-forge Cached + msgpack-python 1.0.5 py311hd6ee22a_0 conda-forge Cached + lz4 4.3.2 py311h854b99e_0 conda-forge Cached + fonttools 4.42.1 py311heffc1b2_0 conda-forge 3MB + cytoolz 0.12.2 py311heffc1b2_0 conda-forge Cached + contourpy 1.1.0 py311he4fd1f5_0 conda-forge Cached + pyarrow 13.0.0 py311hd7bc329_0_cpu conda-forge 4MB + pandas 2.0.3 py311h9e438b8_1 conda-forge Cached + cftime 1.6.2 py311h4add359_1 conda-forge Cached + numcodecs 0.11.0 py311ha397e9f_2 conda-forge Cached + matplotlib-base 3.7.2 py311h3bc9839_0 conda-forge 8MB + netcdf4 1.6.4 nompi_py311h724ebb5_101 conda-forge 444kB + urllib3 2.0.4 pyhd8ed1ab_0 conda-forge 98kB + jinja2 3.1.2 pyhd8ed1ab_1 conda-forge Cached + pymbolic 2022.2 pyhd8ed1ab_0 conda-forge 90kB + cgen 2020.1 py_0 conda-forge 16kB + dask-core 2023.8.1 pyhd8ed1ab_0 conda-forge Cached + xarray 2023.8.0 pyhd8ed1ab_0 conda-forge Cached + zarr 2.16.1 pyhd8ed1ab_0 conda-forge Cached + requests 2.31.0 pyhd8ed1ab_0 conda-forge Cached + bokeh 3.2.2 pyhd8ed1ab_0 conda-forge Cached + distributed 2023.8.1 pyhd8ed1ab_0 conda-forge Cached + pooch 1.7.0 pyha770c72_3 conda-forge Cached + dask 2023.8.1 pyhd8ed1ab_0 conda-forge Cached + scipy 1.11.2 py311h93d07a4_0 conda-forge 15MB + parcels 2.4.2 pyha2b7ab6_2 conda-forge 142kB Summary: Install: 177 packages Total download: 118MB ────────────────────────────────────────────────────────────────────────────────────────── Dry run. Not executing the transaction. ```
You're already using micromamba it seems: https://github.com/OceanParcels/parcels/blob/2bc0d05da8a9ed7a4e70823abac0731be86542c6/.github/actions/install-parcels/action.yml#L21C1-L23C1
Also, no changes are needed in the env files unless you want to use the selectors. I still don't think the selectors are needed, unless I am misunderstanding the issue at play.
Oh wait, you are trying to have dev environments... Okay, for those, I'd keep the separate arch files. There is no way around that. You will need to somehow tell the solver about different needs on different systems. We did that in the parcel feedstock by manipulating the recipe, but you cannot do that easily elsewhere
You will need to somehow tell the solver about different needs on different systems.
You can by using sel(arch)
, for example:
name: TEST
channels:
- conda-forge
dependencies:
- python>=3.8
- sel(unix): mpich>=3.2.1 # will be installed only on not-windows
- sel(win): colorama # will be installed only on windows
Is that what you want @erikvansebille ?
You can by using
sel(arch)
Is that what you want @erikvansebille ?
Thanks for this suggestion, @ocefpaf! It works for Github Action, seehttps://github.com/OceanParcels/parcels/pull/1416/commits/e8be1af07aeb3223cf1b0b7cfcb925d05a486700
However, the build for ReadtheDocs now fails, using the same environment.yml file. See https://readthedocs.org/projects/parcels/builds/21739117/
mamba env create --quiet --name 1416 --file environment.yml
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Unable to install package for sel(unix).
Please double check and ensure your dependencies file has
the correct spelling. You might also try installing the
conda-env-sel(unix) package to see if provides the required
installer.
Retrieving notices: ...working... done
Any idea why the mamba
implementation that readthedocs uses doesn't accept these sel(unix)
decorators? Am I missing something?
Any idea why the
mamba
implementation that readthedocs uses doesn't accept thesesel(unix)
decorators? Am I missing something?
Do you know what version they are using? It may be a matter of asking them to update it.
Do you know what version they are using? It may be a matter of asking them to update it.
It's mambaforge-22.9.0-3, see also the screenshot below from https://readthedocs.org/projects/parcels/builds/21739117/
Is that indeed an older version where sel(unix)
is not implemented yet?
Is that indeed an older version where sel(unix) is not implemented yet?
Mambaforge 22.9.0 uses mamba mamba 1.1.0 but latest is 1.5.0. If I had to guess, yes, that one is too old. But I could not find any mentions of it in the changelog.
I'm not experienced with readthedocs to know for sure but can you control and/or install/update the underlying conda flavor in it?
Latest mambaforge is 23.3.1 BTW.
This is now fixed in #1416
Thank you so much for adding
osx-arm64
support in v2.4.0!We tried creating a conda environment on an Apple M1:
conda create -n test-env -c conda-forge parcels
conda activate test-env
gives the following error:We asked about this error here and they suggested that the
parcels
recipe might be wrong. They linked to https://github.com/conda-forge/parcels-feedstock/blob/main/recipe/meta.yaml#L38.We thought maybe
- clang_osx-arm64
needed to be added to the recipe, but we noticed that there's already a commit that removed- clang_osx-arm64
after adding it.Do you have any further guidance on this issue? Thanks again!