Open alexgleith opened 3 months ago
Hi @alexgleith, nice to see those Rust geomedian algorithms coming along!
I'm trying to see what the best approach here would be. We probably don't want to install all those Linux compilers explicitly in base-image
, given that we're trying to move towards cross-platform builds (e.g. @scottyhq's comment on having osx-arm64 builds at https://github.com/pangeo-data/pangeo-docker-images/issues/396#issuecomment-2148040753).
We could just depend on rust
itself, and let conda/mamba pull in the necessary compilers for each platform. But looking at https://anaconda.org/conda-forge/rust/files?version=1.77.2, the Rust package itself is 100-200MB in size, and there will be more MBs added for the compilers, so not sure if this is a good idea...
That said, we could consider making a base-rust
image perhaps? The packaging situation for Rust is a little different since most people just use cargo
to install crates rather than conda/pip, so we'd need to think this through a bit.
Hey @weiji14
This is for my own Docker Image, so I don't really mind how big it is, at least for now. I'd just like a solution that works! I don't really want to have an external build process for this Python library, though admittedly, it does make sense to remove the compiler and rust dependencies...
Anyhow, this should be possible to do, right? I can't understand why I can't install GCC and Rust and compile the package!
Ah ok, I thought you were asking for a solution on pangeo-docker-images
to make this work out of the box :sweat_smile:
Could you try some of the suggestions at https://github.com/conda-forge/rust-feedstock/issues/71? I see something about setting a RUSTFLAGS
environment variable, though that should be taken care of if you have activated the conda environment correctly.
That looks like a good option! Testing now...
Doesn't seem to have helped... I'll do some testing locally in a built Docker image, rather than in CI.
Testing in a running Docker image I get the below:
env | grep RUST
RUSTFLAGS=-C link-arg=-Wl,-rpath-link,/srv/conda/envs/notebook/lib -C link-arg=-Wl,-rpath,/srv/conda/envs/notebook/lib -L /srv/conda/envs/notebook/lib
python -m pip install git+https://github.com/opendatacube/odc-algo.git@add-rust-geomedian-impl
Results in a huge list of errors including:
note: /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc: No such file or directory
To narrow down that possible issues a bit, if you just use an environment.yml with the micromamba docker image https://micromamba-docker.readthedocs.io/en/latest/quick_start.html does it work?
Describe the bug I'm trying to install a Python package from GitHub that needs some rust components compiled. I've added
To Reproduce Steps to reproduce the behavior:
Add these to the environment.yaml:
Add this to Python requirements.txt
git+https://github.com/opendatacube/odc-algo.git@add-rust-geomedian-impl
Expected behavior
The Python package builds and installs in the Conda environment.
Docker Image Version (e.g. quay.io/pangeo/ml-notebook:2023.02.27):
FROM pangeo/base-image:2024.03.22
Infrastructure (Where you are running this image):
Error output is this: