secondlaw-ai / skyrim

🌎 🤝 AI weather models united
Apache License 2.0
150 stars 7 forks source link

ecCodes library not found from earth2mip when using the Pangu model #27

Closed christegho closed 1 month ago

christegho commented 1 month ago

I am trying to use the Pangu model:

model = Skyrim("pangu", ic_source="gfs")

and so far I gathered I need to install earth2mip with python3.10.

However, I get the following error:

/usr/local/lib/python3.10/site-packages/ecmwflibs/__init__.py:81: UserWarning: libglib-2.0.so.0: cannot open shared object file: No such file or directory
  warnings.warn(str(e))
Traceback (most recent call last):
  File "/pkg/modal/_container_io_manager.py", line 733, in handle_input_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 382, in run_input_sync
    res = io_context.call_finalized_function()
  File "/pkg/modal/_container_io_manager.py", line 166, in call_finalized_function
    res = self.finalized_function.callable(*args, **kwargs)
  File "/root/test.py", line 18, in get_pangu_forecast
    from skyrim.core import Skyrim
  File "/usr/local/lib/python3.10/site-packages/skyrim/core/__init__.py", line 2, in <module>
    from .skyrim import Skyrim
  File "/usr/local/lib/python3.10/site-packages/skyrim/core/skyrim.py", line 6, in <module>
    from .models import MODELS
  File "/usr/local/lib/python3.10/site-packages/skyrim/core/models/__init__.py", line 1, in <module>
    from .pangu import PanguModel
  File "/usr/local/lib/python3.10/site-packages/skyrim/core/models/pangu.py", line 3, in <module>
    from .base import GlobalModel
  File "/usr/local/lib/python3.10/site-packages/skyrim/core/models/base.py", line 8, in <module>
    from ...libs.ic import get_data_source
  File "/usr/local/lib/python3.10/site-packages/skyrim/libs/ic/__init__.py", line 1, in <module>
    from earth2mip.initial_conditions import get_data_source as get_data_source_e2m, base
  File "/usr/local/lib/python3.10/site-packages/earth2mip/initial_conditions/__init__.py", line 24, in <module>
    from earth2mip.initial_conditions import base, cds, gfs, hdf5, hrmip, ifs
  File "/usr/local/lib/python3.10/site-packages/earth2mip/initial_conditions/cds.py", line 26, in <module>
    import eccodes
  File "/usr/local/lib/python3.10/site-packages/eccodes/__init__.py", line 13, in <module>
    from .eccodes import *  # noqa
  File "/usr/local/lib/python3.10/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import (
  File "/usr/local/lib/python3.10/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/usr/local/lib/python3.10/site-packages/gribapi/gribapi.py", line 34, in <module>
    from gribapi.errors import GribInternalError
  File "/usr/local/lib/python3.10/site-packages/gribapi/errors.py", line 16, in <module>
    from .bindings import ENC, ffi, lib
  File "/usr/local/lib/python3.10/site-packages/gribapi/bindings.py", line 35, in <module>
    raise RuntimeError("Cannot find the ecCodes library")
RuntimeError: Cannot find the ecCodes library

Have you seen this error before?

Notes, I am using modal and I have the following image and function deployed:

import modal

app = modal.App("pangu")

skyrim_image = (
    modal.Image.debian_slim(python_version="3.10")
    .apt_install("git")
     .pip_install("ecmwf-opendata", "python-dotenv", "onnxruntime", "eccodes")
    .run_commands(   
        "git clone https://github.com/secondlaw-ai/skyrim.git && cd skyrim && pip install .",
        "git clone https://github.com/NVIDIA/earth2mip.git && cd earth2mip && pip install ."

    )
)

@app.function(gpu="A10G", image=skyrim_image)
def get_pangu_forecast(date: str):
    from skyrim.core import Skyrim
    import loguru
    import warnings
    warnings.filterwarnings("ignore")
    loguru.logger.disable("skyrim")

    model = Skyrim("pangu", ic_source="gfs")

    final_pred, pred_paths = model.predict(
        date=date, time="0000", lead_time=24 * 7, save=True
    )
    return final_pred, pred_paths
christegho commented 1 month ago

I resolved this via sudo apt-get install -y libeccodes-dev on linux and brew install eccodes on mac