Replace singularity with apptainer

sverhoeven commented 1 year ago

Apptainer installation ships with symlink to /usr/bin/singularity, so creating models with BmiClientSingularity still works.
Depending on ewatercylce python package version a slightly different config file is generated.

TODO

[x] check apptainer deps are installed, see https://github.com/eWaterCycle/setup-apptainer/issues/21

Testable by

spinning up a VM in SRC or with vagrant locally
Login to JupyterHub
Open a terminal
Run apptainer run docker://ghcr.io/apptainer/lolcow, it should produce ascii art of a cow when apptainer is installed correctly.
Run ls -l $(which singularity), singularity should be symbolic link to apptainer.
Run pip show ewatercycle should have version 1.4.1 installed
Verify /etc/ewatercycle.yaml has container_engine: apptainer.

sverhoeven commented 1 year ago

on SRC creation failed

```shell TASK [Run Ansible plugin] ****************************************************** Monday 05 June 2023 11:27:46 +0000 (0:00:01.196) 0:04:07.036 *********** fatal: [145.38.187.131]: FAILED! => {"ansible_job_id": "495034654240.12475", "changed": true, "cmd": ["ansible-playbook", "--connection=local", "-b", "--extra-vars={'alt_home_location': '/data/volume_2', 'dcache_ro_token': '***', 'rclone_cache_dir': '/data/volume_3', 'rclone_max_gsize': '45'}", "/rsc/plugins/39795e81-c3e0-408b-9e80-a936c42048d1/research-cloud-plugin.yml"], "delta": "0:00:00.671938", "end": "2023-06-05 11:27:48.058763", "finished": 1, "msg": "non-zero return code", "rc": 2, "start": "2023-06-05 11:27:47.386825", "stderr": "[WARNING]: No inventory was parsed, only implicit localhost is available\n[WARNING]: provided hosts list is empty, only localhost is available. Note that\nthe implicit localhost does not match 'all'\n[WARNING]: Reset is not implemented for this connection", "stderr_lines": ["[WARNING]: No inventory was parsed, only implicit localhost is available", "[WARNING]: provided hosts list is empty, only localhost is available. Note that", "the implicit localhost does not match 'all'", "[WARNING]: Reset is not implemented for this connection"], "stdout": "\nPLAY [Install and configure eWaterCycle Jupyter] *******************************\n\nTASK [Wait for system to become reachable] *************************************\nok: [localhost]\n\nTASK [Gather facts for first time] *********************************************\nfatal: [localhost]: FAILED! => {\"changed\": false, \"module_stderr\": \"/bin/sh: 1: powershell: not found\\n\", \"module_stdout\": \"\", \"msg\": \"The module failed to execute correctly, you probably need to set the interpreter.\\nSee stdout/stderr for the exact error\", \"rc\": 127}\n\nPLAY RECAP *********************************************************************\nlocalhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 ", "stdout_lines": ["", "PLAY [Install and configure eWaterCycle Jupyter] *******************************", "", "TASK [Wait for system to become reachable] *************************************", "ok: [localhost]", "", "TASK [Gather facts for first time] *********************************************", "fatal: [localhost]: FAILED! => {\"changed\": false, \"module_stderr\": \"/bin/sh: 1: powershell: not found\\n\", \"module_stdout\": \"\", \"msg\": \"The module failed to execute correctly, you probably need to set the interpreter.\\nSee stdout/stderr for the exact error\", \"rc\": 127}", "", "PLAY RECAP *********************************************************************", "localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 "]} ```

Peter9192 commented 1 year ago

vagrant@default:~$ apptainer run docker://ghcr.io/apptainer/lolcow
INFO:    Using cached SIF image
INFO:    underlay of /etc/localtime required more than 50 (77) bind mounts
 ______________________________
< Wed Jun 14 12:58:23 UTC 2023 >
 ------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Peter9192 commented 1 year ago

Apptainer works. However when trying to run a generated notebook I get


ModuleNotFoundError                       Traceback (most recent call last)
File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:9
      8 try:
----> 9     import ESMF as esmpy  # noqa: N811
     10 except ImportError:

ModuleNotFoundError: No module named 'ESMF'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 import ewatercycle.parameter_sets
      2 import ewatercycle.forcing
      3 import ewatercycle.models

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/__init__.py:1
----> 1 from .config import CFG
      2 from .version import __version__

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/config/__init__.py:87
      1 """Config
      2 ******
      3 
   (...)
     84     wflow.docker_images: ewatercycle/wflow-grpc4bmi:2020.1.1
     85 """
---> 87 from ._config_object import CFG, DEFAULT_CONFIG, SYSTEM_CONFIG, USER_HOME_CONFIG, Config
     89 __all__ = ["CFG", "Config", "DEFAULT_CONFIG", "SYSTEM_CONFIG", "USER_HOME_CONFIG"]

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/config/_config_object.py:11
      7 from typing import Optional, TextIO, Union
      9 from ruamel.yaml import YAML
---> 11 from ewatercycle.util import to_absolute_path
     13 from ._validated_config import ValidatedConfig
     14 from ._validators import _validators

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/ewatercycle/util.py:9
      7 import xarray as xr
      8 from dateutil.parser import parse
----> 9 from esmvalcore.experimental.recipe_output import RecipeOutput
     10 from shapely import geometry
     13 def find_closest_point(
     14     grid_longitudes: Iterable[float],
     15     grid_latitudes: Iterable[float],
     16     point_longitude: float,
     17     point_latitude: float,
     18 ) -> Tuple[int, int]:

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/experimental/__init__.py:9
      6 from esmvalcore.config import CFG
      8 from ._warnings import warnings  # prints experimental API warning
----> 9 from .recipe import Recipe
     10 from .utils import RecipeList, get_all_recipes, get_recipe
     12 logging.basicConfig(format='%(message)s',
     13                     level=logging.INFO,
     14                     stream=sys.stdout)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/experimental/recipe.py:12
      8 from typing import Dict, Optional
     10 import yaml
---> 12 from esmvalcore._recipe.recipe import Recipe as RecipeEngine
     13 from esmvalcore.config import CFG, Session
     15 from ._logging import log_to_dir

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/_recipe/recipe.py:24
     22 from esmvalcore.config._config import TASKSEP, get_project_config
     23 from esmvalcore.config._diagnostics import TAGS
---> 24 from esmvalcore.dataset import Dataset
     25 from esmvalcore.exceptions import (
     26     ESMValCoreDeprecationWarning,
     27     InputFilesNotFound,
     28     RecipeError,
     29 )
     30 from esmvalcore.local import (
     31     _dates_to_timerange,
     32     _get_multiproduct_filename,
   (...)
     35     _truncate_dates,
     36 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/dataset.py:18
     15 from iris.cube import Cube
     17 from esmvalcore import esgf, local
---> 18 from esmvalcore._recipe import check
     19 from esmvalcore._recipe.from_datasets import datasets_to_recipe
     20 from esmvalcore.cmor.table import _get_mips, _update_cmor_facets

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/_recipe/check.py:17
     15 from esmvalcore.exceptions import InputFilesNotFound, RecipeError
     16 from esmvalcore.local import _get_start_end_year, _parse_period
---> 17 from esmvalcore.preprocessor import TIME_PREPROCESSORS, PreprocessingTask
     18 from esmvalcore.preprocessor._multimodel import STATISTIC_MAPPING
     19 from esmvalcore.preprocessor._supplementary_vars import (
     20     PREPROCESSOR_SUPPLEMENTARIES,
     21 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py:27
     25 from ._bias import bias
     26 from ._cycles import amplitude
---> 27 from ._derive import derive
     28 from ._detrend import detrend
     29 from ._io import (
     30     _get_debug_filename,
     31     cleanup,
   (...)
     35     write_metadata,
     36 )

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/__init__.py:32
     28         derivers[short_name] = getattr(module, 'DerivedVariable')
     29     return derivers
---> 32 ALL_DERIVED_VARIABLES = _get_all_derived_variables()
     34 __all__ = list(ALL_DERIVED_VARIABLES)
     37 def get_required(short_name, project):

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/__init__.py:26, in _get_all_derived_variables()
     24 for path in Path(__file__).parent.glob('[a-z]*.py'):
     25     short_name = path.stem
---> 26     module = importlib.import_module(
     27         f'esmvalcore.preprocessor._derive.{short_name}')
     28     derivers[short_name] = getattr(module, 'DerivedVariable')
     29 return derivers

File /opt/conda/envs/ewatercycle/lib/python3.11/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_derive/sispeed.py:6
      3 import logging
      4 from iris import Constraint
----> 6 from .._regrid import regrid
      8 from ._baseclass import DerivedVariableBase
     10 logger = logging.getLogger(__name__)

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid.py:24
     22 from ..cmor._fixes.shared import add_altitude_from_plev, add_plev_from_altitude
     23 from ..cmor.table import CMOR_TABLES
---> 24 from ._regrid_esmpy import ESMF_REGRID_METHODS
     25 from ._regrid_esmpy import regrid as esmpy_regrid
     26 from ._supplementary_vars import add_ancillary_variable, add_cell_measure

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:11
      9         import ESMF as esmpy  # noqa: N811
     10     except ImportError:
---> 11         raise exc
     12 import iris
     13 import numpy as np

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py:5
      2 """Provides regridding for irregular grids."""
      4 try:
----> 5     import esmpy
      6 except ImportError as exc:
      7     # Prior to v8.4.0, `esmpy`` could be imported as `ESMF`.
      8     try:

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/__init__.py:112
    108 __obsoletes__ = msg["obsoletes"]
    110 #### IMPORT LIBRARIES #########################################################
--> 112 from esmpy.api.esmpymanager import *
    113 from esmpy.api.grid import *
    114 from esmpy.api.mesh import *

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/api/esmpymanager.py:9
      3 """
      4 The ESMPyManager API
      5 """
      7 #### IMPORT LIBRARIES #########################################################
----> 9 from esmpy.interface.cbindings import *
     11 from esmpy.api.constants import *
     12 from esmpy.util.exceptions import *

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/interface/cbindings.py:13
     11 import esmpy.api.constants as constants
     12 from esmpy.util.decorators import *
---> 13 from esmpy.interface.loadESMF import _ESMF
     16 def copy_struct(src):
     17     dst = type(src)()

File /opt/conda/envs/ewatercycle/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:28
     26     esmfmk = os.environ["ESMFMKFILE"]
     27 except:
---> 28     raise ImportError('The ESMFMKFILE environment variable is not available.')
     30 #### INVESTIGATE esmf.mk ######################################################
     31 
     32 # TODO: look for various dependecies in the ESMF build log
   (...)
     37 #       use this information to set variables that can be checked at beginning
     38 #       of the routines that require an ESMF build with these dependencies
     40 with open(esmfmk, 'r') as MKFILE:
     41     
     42     # investigate esmf.mk

ImportError: The ESMFMKFILE environment variable is not available.

Peter9192 commented 1 year ago

Lot's of issues on ESMValCore and Iris about this, e.g.

https://github.com/ESMValGroup/ESMValCore/pull/1876 https://github.com/SciTools-incubator/iris-esmf-regrid/pull/231

eWaterCycle / infra

Replace singularity with apptainer #137