Closed priscavdsluis closed 5 months ago
Describe bug:
failed to install dfm_tools
on Mac OSX
OS
Darwin Bjorns-MacBook-Pro-2.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Dec 19 20:43:09 PST 2022; root:xnu-8020.240.18~2/RELEASE_ARM64_T6000 arm64
conda --version
conda 23.1.0
pip --version
pip 23.0.1 from /Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip (python 3.8)
Minimal example
conda create --name dfm_tools_env -c conda-forge python=3.8
conda activate dfm_tools_env
conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
python -m pip install git+https://github.com/Deltares/dfm_tools
Error message on exactuting python -m pip install git+https://github.com/Deltares/dfm_tools
Collecting pydantic<1.11,>=1.10
Using cached pydantic-1.10.6-cp38-cp38-macosx_11_0_arm64.whl (2.5 MB)
Collecting meshkernel<3.0.0,>=2.0.0
Using cached meshkernel-2.0.2.tar.gz (364 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Traceback (most recent call last):
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
self.run_setup()
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 484, in run_setup
super(_BuildMetaLegacyBackend,
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-40rtvd7u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "<string>", line 222, in <module>
File "<string>", line 70, in get_library_name
OSError: Unsupported operating system: Darwin
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Additional context
Using terminal in Visual Studio Code
Version: 1.76.2 (Universal)
Commit: ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
Date: 2023-03-14T17:54:09.061Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Darwin arm64 21.6.0
Sandboxed: No
Describe the issue:
failed to install dfm_tools
on Linux
conda --version conda 4.11.0
pip --version pip 21.3.1 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9)
Minimal example
conda create --name dfm_tools_env -c conda-forge python=3.8
conda activate dfm_tools_env
conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
python -m pip install git+https://github.com/Deltares/dfm_tools
Error message on executing python -m pip install git+https://github.com/Deltares/dfm_tools
Building wheels for collected packages: dfm-tools, meshkernel
Building wheel for dfm-tools (setup.py) ... done
Created wheel for dfm-tools: filename=dfm_tools-0.10.31-py3-none-any.whl size=70826 sha256=0aa11eeb1e435ae56b697bf96ad9eff595c537bba29fb7bb24f8f6a574624d9f
Stored in directory: /tmp/pip-ephem-wheel-cache-22p4a3ww/wheels/e7/9f/34/3e224a6d2ec93d9c662c3188c7eda64aea08650ea083ad224d
Building wheel for meshkernel (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for meshkernel (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [19 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-38
creating build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/factories.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/meshkernel.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/py_structures.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/utils.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/c_structures.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/__init__.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/version.py -> build/lib.linux-aarch64-cpython-38/meshkernel
copying meshkernel/errors.py -> build/lib.linux-aarch64-cpython-38/meshkernel
running build_ext
git clone https://github.com/Deltares/MeshKernel
Cloning into 'MeshKernel'...
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DADD_UNIT_TESTS_PROJECTS=OFF
error: command 'cmake' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for meshkernel
Successfully built dfm-tools
Failed to build meshkernel
ERROR: Could not build wheels for meshkernel, which is required to install pyproject.toml-based projects
Additional context
Using terminal in JupyterLab
Version 3.2.6
OS
Linux e1dd4ba4d0ab 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Meshkernel is available as a compiled library in the wheel packages for both Windows and Linux. To install it, simply run the command 'pip install meshkernel'. There's no need to build Meshkernel from sources for these operating systems.
However, for Mac, there is currently no solution available as we don't have macOS agents. The Meshkernel 1.0.0 wheel package did include a compiled macOS library that was generated by one of our developers on their laptop.
Thanks @lucacarniato
So you're saying I can install dfm_tools
by doing:
conda create --name dfm_tools_env -c conda-forge python=3.8
conda activate dfm_tools_env
conda install -c conda-forge git shapely cartopy pyepsg geopandas contextily xarray dask netcdf4 bottleneck xugrid cdsapi pydap
pip install meshkernel
python -m pip install git+https://github.com/Deltares/dfm_tools
installing meshkernel before dfm_tools could work
Hi @lucacarniato,
When I follow the steps in https://github.com/Deltares/HYDROLIB-core/issues/444#issuecomment-1488542448 on a Mac, I get this error:
(dfm_tools_env) backeb@Bjorns-MacBook-Pro-2:bed-data-preparation>pip install meshkernel
Collecting meshkernel
Using cached meshkernel-2.0.2.tar.gz (364 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Traceback (most recent call last):
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/backeb/mambaforge/envs/dfm_tools_env/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
self.run_setup()
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
super(_BuildMetaLegacyBackend,
File "/private/var/folders/0d/d0d2xvbs4p1061xfbpnzrt040000gn/T/pip-build-env-sjs_ibft/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
exec(code, locals())
File "<string>", line 222, in <module>
File "<string>", line 70, in get_library_name
OSError: Unsupported operating system: Darwin
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Will test on Linux now.
Following this approach on a CentOS 8 VM, I managed to install dfm_tools
.
Hi @backeb, regarding your original error: "OSError: Unsupported operating system: Darwin", indeed, the latest MeshKernel(Py) does not come in a MacOSX version. I've already listed it in the hydrolib-core release notes from previous month: https://deltares.github.io/HYDROLIB-core/0.5.2/releasenotes/#compatibility-notes But maybe @lucacarniato can list it somewhere on the meshkernel pages as well.
Thanks @arthurvd, I'm working on a docker container, so I can use dfm_tools from my Mac.
@arthurvd I've tried some steps to getting it working in an ubuntu docker container here: https://github.com/openearth/bed-data-preparation/issues/1#issuecomment-1545822906 but the install fails on a cmake
dependency.
Meshkernelpy 2.1.0 was released a few weeks ago, macos/linux support was fixed here. This issue can probably be fixed by updating the dependencies.
Will be fixed with:
In the next release of meshkernel
If macOS support is fixed, we should re-add the macOS in our GitHub ci.yml workflow.
Update after adding macos to the github workflow:
Installation succeeds, but there is one failing testcase on macos-latest (14) even though macos-13 testbench is all green:
FAILED tests/dflowfm/test_net.py::test_create_1d_2d_1d2d
This is the plain HYDROLIB-core code, the test fails on assert network1_link1d2d.shape == (21, 2)
:
import numpy as np
from meshkernel import GeometryList
from hydrolib.core.dflowfm.net.models import Branch, Network
def get_circle_gl(r, detail=100):
t = np.r_[np.linspace(0, 2 * np.pi, detail), 0]
polygon = GeometryList(np.cos(t) * r, np.sin(t) * r)
return polygon
# Define line (spiral)
theta = np.arange(0.1, 20, 0.01)
y = np.sin(theta) * theta
x = np.cos(theta) * theta
dists = np.r_[0.0, np.cumsum(np.hypot(np.diff(x), np.diff(y)))]
dists = dists[np.arange(0, len(dists), 20)]
# Create branch
branch = Branch(geometry=np.stack([x, y], axis=1), branch_offsets=dists)
# Create Mesh1d
network = Network()
network.mesh1d_add_branch(branch, name="branch1")
branch = Branch(geometry=np.array([[-25.0, 0.0], [x[0], y[0]]]))
branch.generate_nodes(mesh1d_edge_length=2.5)
network.mesh1d_add_branch(branch, name="branch2")
# Add Mesh2d
network.mesh2d_create_rectilinear_within_extent(
extent=(-22, -22, 22, 22), dx=2, dy=2
)
network.mesh2d_clip_mesh(geometrylist=get_circle_gl(22))
network.mesh2d_refine_mesh(polygon=get_circle_gl(11), level=1)
network.mesh2d_refine_mesh(polygon=get_circle_gl(3), level=1)
# Add links
network.link1d2d_from_1d_to_2d(branchids=["branch1"], polygon=get_circle_gl(19))
mesh2d_output = network._mesh2d.get_mesh2d()
assert len(mesh2d_output.face_x) == 152
mesh1d_output = network._mesh1d._get_mesh1d()
assert len(mesh1d_output.node_x) == 110
network1_link1d2d = network._link1d2d.link1d2d
network1_con_m1d = network._link1d2d.meshkernel.contacts_get().mesh1d_indices
network1_con_m2d = network._link1d2d.meshkernel.contacts_get().mesh2d_indices
assert network1_link1d2d.shape == (21, 2)
assert network1_con_m1d.size == 21
assert network1_con_m2d.size == 21
Also, could you please install meshkernel in a clean environment with pip install "meshkernel>=4.1.0"
and let me know if this works? Also for HYDROLIB-core, please try both pip install hydrolib-core
and pip install git+https://github.com/deltares/hydrolib-core
. All in a separate clean environment and let me know if it installs and what versions you get.
The installation fails because of an issue with installing meshkernelpy: https://github.com/Deltares/MeshKernelPy/issues/162
Therefore I cannot run the test in your code yet.
After some trial and error I noticed that the macos-12
and macos-13
testbanks do run (both seem to be x86_64), so the failing testcase seems to be macos-14
(arm64):
Also, the installation issues by @backeb above seem to be meshkernel related, since it does not cover macos-13-arm64 since we do not have these runners available. A meshkernelpy issues was created: https://github.com/Deltares/MeshKernelPy/issues/162
Therefore, I suggest to wrap up this issue (which goal is to repair the linux and macos testbenches) by testing on macos-13 now. I have created a follow-up issue to investigate macos 14 arm64 support here: https://github.com/Deltares/HYDROLIB-core/issues/635
What is the need for this task. We are trying to upgrade MeshKernel from 1.0.0 to 2.0.0, see PR #442. When doing that we notice that the CI is failing on Linux and MacOS:
What is the task? Investigate why these tests are failing and whether it is on the hydrolib-core or meshkernel side.
Additional context The stacktrace: stacktraces.txt