search

OGGM / oggm

Open Global Glacier Model
http://oggm.org
BSD 3-Clause "New" or "Revised" License
224 stars 105 forks source link

OGGM installation issue (test_prepro, test_shop, test_utils) #1498

Closed xuepingFu closed 1 year ago

xuepingFu commented 2 years ago

When I installed OGGM on Ubuntu 20.04, I get some following errors and how can I fix this errors? My test result are as following:

(oggm_env1) fuxp@fuxp-virtual-machine:~$ pytest --pyargs oggm =========================================================================================== test session starts =========================================================================================== platform linux -- Python 3.9.7, pytest-7.1.3, pluggy-1.0.0 Matplotlib: 3.6.1 Freetype: 2.12.1 rootdir: /home/fuxp plugins: mpl-0.150.0, anyio-3.6.1 collected 373 items

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_benchmarks.py ...s.ss [ 1%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py .ss.......s...s....sss [ 7%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_minimal.py ... [ 8%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_models.py ................................ssss.......ssss.ss.ss..ssssssssss..ssssssssssssssssssssssssssssssss.s [ 35%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_numerics.py sssss.ss.sssss.s.sssss.sss.sss.s [ 44%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py .................s..............s....ss...ss..s..F..ss.....sssssss..sssss....s....... [ 67%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_shop.py ss....F.....s. [ 70%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_utils.py ...........................sss.F.....sss...s....................ss.s..ssss.ssssss..sss...s.ss.ss.ss.... [ 98%] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_workflow.py ssssss [100%]

================================================================================================ FAILURES ================================================================================================= ____ TestClimate.test_ref_mbglaciers ____

self = 

def test_ref_mb_glaciers(self):

    hef_file = get_demo_file('Hintereisferner_RGI5.shp')
    entity = gpd.read_file(hef_file).iloc[0]
    gdir = oggm.GlacierDirectory(entity, base_dir=self.testdir)

    rids = utils.get_ref_mb_glaciers_candidates()
    assert len(rids) > 200

    rids = utils.get_ref_mb_glaciers_candidates(gdir.rgi_version)
    assert len(rids) > 200

    assert len(cfg.DATA) >= 2

    with pytest.raises(InvalidWorkflowError):
      utils.get_ref_mb_glaciers([gdir])

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py:2096:

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:550: in _global_task return task_func(gdirs, *kwargs) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:617: in get_ref_mb_glaciers mbdf = g.get_ref_mb_data(y0=y0, y1=y1) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:3454: in get_ref_mb_data ci = self.get_climate_info(input_filesuffix=input_filesuffix) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:3149: in get_climate_info with ncDataset(f) as nc: anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:361: in init super(ncDataset, self).init(args, **kwargs) src/netCDF4/_netCDF4.pyx:2353: in netCDF4._netCDF4.Dataset.init ???

??? E UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)

src/netCDF4/_netCDF4.pyx:1961: UnicodeDecodeError ------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------- 2022-10-17 17:14:35: oggm.cfg: Reading default parameters from the OGGM params.cfg configuration file. 2022-10-17 17:14:35: oggm.cfg: Multiprocessing switched OFF according to the parameter file. 2022-10-17 17:14:35: oggm.cfg: Multiprocessing: using all available processors (N=4) 2022-10-17 17:14:35: oggm.cfg: PARAMS['border'] changed from 40 to 10. 2022-10-17 17:14:35: oggm.cfg: PARAMS['baseline_climate'] changed from GSWP3_W5E5 to `. 2022-10-17 17:14:35: oggm.cfg: PARAMS['use_tstar_calibration'] changed fromFalsetoTrue. 2022-10-17 17:14:35: oggm.cfg: PARAMS['use_winter_prcp_factor'] changed fromTruetoFalse. 2022-10-17 17:14:35: oggm.cfg: WARNING: adding an unknown parameterprcp_scaling_factor:2.5to PARAMS. 2022-10-17 17:14:35: oggm.cfg: PARAMS['hydro_month_nh'] changed from1to10. 2022-10-17 17:14:35: oggm.cfg: PARAMS['hydro_month_sh'] changed from1to4`. 2022-10-17 17:14:35: oggm.utils: Applying global task get_ref_mbglaciers on 1 glaciers ____ Test_w5e5.test_process_gswp3_w5e5_data __

self = <oggm.tests.test_shop.Test_w5e5 object at 0x7fe7752af970>, class_case_dir = '/home/fuxp/OGGM/tests/a3dd618743ed9b57677e1619413c726d/Test_w5e5'

def test_process_gswp3_w5e5_data(self, class_case_dir):

    # Init
    cfg.initialize()
    cfg.PARAMS['use_intersects'] = False
    cfg.PATHS['working_dir'] = class_case_dir
    cfg.PATHS['dem_file'] = get_demo_file('hef_srtm.tif')
    hef_file = get_demo_file('Hintereisferner_RGI5.shp')
    cfg.PARAMS['hydro_month_nh'] = 1
    gdir = workflow.init_glacier_directories(gpd.read_file(hef_file))[0]
    from oggm.shop import w5e5
    w5e5.process_gswp3_w5e5_data(gdir)
    path_clim = gdir.get_filepath('climate_historical')

    ds_clim = xr.open_dataset(path_clim)
  assert ds_clim.time[0]['time.year'] == 1901

E AssertionError: assert <xarray.DataArray 'year' ()>\narray(1979)\nCoordinates:\n time datetime64[ns] 1979-01-01 == 1901

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_shop.py:276: AssertionError ------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------- 2022-10-17 17:15:31: oggm.cfg: Reading default parameters from the OGGM params.cfg configuration file. 2022-10-17 17:15:31: oggm.cfg: Multiprocessing switched OFF according to the parameter file. 2022-10-17 17:15:31: oggm.cfg: Multiprocessing: using all available processors (N=4) 2022-10-17 17:15:31: oggm.cfg: PARAMS['use_intersects'] changed from True to False. 2022-10-17 17:15:31: oggm.workflow: Execute entity tasks [GlacierDirectory] on 1 glaciers 2022-10-17 17:15:31: oggm.shop.w5e5: (RGI50-11.00897) process_gswp3_w5e5_data 2022-10-17 17:15:31: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_climate/gswp3-w5e5/flattened/monthly/gswp3-w5e5_obsclim_tas_global_monthly_1901_2019_flat_glaciers.nc 2022-10-17 17:15:31: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_climate/gswp3-w5e5/flattened/monthly/gswp3-w5e5_obsclim_pr_global_monthly_1901_2019_flat_glaciers.nc 2022-10-17 17:15:31: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_climate/gswp3-w5e5/flattened/monthly/gswp3-w5e5_glacier_invariant_flat.nc 2022-10-17 17:15:31: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_climate/gswp3-w5e5/flattened/monthly/gswp3-w5e5_obsclim_temp_std_global_monthly_1901_2019_flat_glaciers.nc __ TestStartFromOnlinePrepro.test_corrupted_file __

self = 

def test_corrupted_file(self):

    # Go - initialize working directories
    gdirs = workflow.init_glacier_directories(['RGI60-11.00787'],
                                              prepro_base_url=TEST_GDIR_URL,
                                              from_prepro_level=4,
                                              prepro_rgi_version='61',
                                              prepro_border=20)

    cfile = utils.get_prepro_gdir('61', 'RGI60-11.00787', 20, 4,
                                  base_url=TEST_GDIR_URL)
    assert 'cluster.klima.uni-bremen.de/~oggm/' in cfile

    # Replace with a dummy file
    os.remove(cfile)
    with open(cfile, 'w') as f:
        f.write('ups')

    # Since we already verified this will error
    with pytest.raises(tarfile.ReadError):
        workflow.init_glacier_directories(['RGI60-11.00787'],
                                          prepro_base_url=TEST_GDIR_URL,
                                          from_prepro_level=4,
                                          prepro_rgi_version='61',
                                          prepro_border=20)

    # This should retrigger a download and just work
    cfg.DL_VERIFIED.clear()
  gdirs = workflow.init_glacier_directories(['RGI60-11.00787'],

prepro_base_url=TEST_GDIR_URL, from_prepro_level=4, prepro_rgi_version='61', prepro_border=20)

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_utils.py:808:

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:402: in init_glacier_directories gdirs = execute_entity_task(gdir_from_prepro, entities, anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:191: in execute_entity_task out = [pc(gdir) for gdir in gdirs] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:191: in out = [pc(gdir) for gdir in gdirs] anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:108: in call res = self._call_internal(func, arg, kwargs) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:102: in _call_internal return call_func(gdir, **kwargs) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/workflow.py:251: in gdir_from_prepro return oggm.GlacierDirectory(entity, from_tar=from_tar) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:2559: in init robust_tar_extract(from_tar, _dir, delete_tar=delete_tar) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:2444: in robust_tar_extract _back_up_retry(func, RuntimeError) anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:2399: in _back_up_retry return func() anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:2439: in func with tarfile.open(base_tar, 'r') as tf:

cls = <class 'tarfile.TarFile'>, name = '/home/fuxp/OGGM/tests/a3dd618743ed9b57677e1619413c726d/dl_cache/cluster.klima.uni-bremen.de/~oggm/test_gdirs/oggm_v1.1/RGI61/b_020/L4/RGI60-11/RGI60-11.00.tar' mode = 'r', fileobj = None, bufsize = 10240, kwargs = {}, not_compressed = <function TarFile.open..not_compressed at 0x7fe77468b0d0>, comptype = 'tar' func = <bound method TarFile.taropen of <class 'tarfile.TarFile'>>

@classmethod
def open(cls, name=None, mode="r", fileobj=None, bufsize=RECORDSIZE, **kwargs):
    """Open a tar archive for reading, writing or appending. Return
       an appropriate TarFile class.

       mode:
       'r' or 'r:*' open for reading with transparent compression
       'r:'         open for reading exclusively uncompressed
       'r:gz'       open for reading with gzip compression
       'r:bz2'      open for reading with bzip2 compression
       'r:xz'       open for reading with lzma compression
       'a' or 'a:'  open for appending, creating the file if necessary
       'w' or 'w:'  open for writing without compression
       'w:gz'       open for writing with gzip compression
       'w:bz2'      open for writing with bzip2 compression
       'w:xz'       open for writing with lzma compression

       'x' or 'x:'  create a tarfile exclusively without compression, raise
                    an exception if the file is already created
       'x:gz'       create a gzip compressed tarfile, raise an exception
                    if the file is already created
       'x:bz2'      create a bzip2 compressed tarfile, raise an exception
                    if the file is already created
       'x:xz'       create an lzma compressed tarfile, raise an exception
                    if the file is already created

       'r|*'        open a stream of tar blocks with transparent compression
       'r|'         open an uncompressed stream of tar blocks for reading
       'r|gz'       open a gzip compressed stream of tar blocks
       'r|bz2'      open a bzip2 compressed stream of tar blocks
       'r|xz'       open an lzma compressed stream of tar blocks
       'w|'         open an uncompressed stream for writing
       'w|gz'       open a gzip compressed stream for writing
       'w|bz2'      open a bzip2 compressed stream for writing
       'w|xz'       open an lzma compressed stream for writing
    """

    if not name and not fileobj:
        raise ValueError("nothing to open")

    if mode in ("r", "r:*"):
        # Find out which *open() is appropriate for opening the file.
        def not_compressed(comptype):
            return cls.OPEN_METH[comptype] == 'taropen'
        for comptype in sorted(cls.OPEN_METH, key=not_compressed):
            func = getattr(cls, cls.OPEN_METH[comptype])
            if fileobj is not None:
                saved_pos = fileobj.tell()
            try:
                return func(name, "r", fileobj, **kwargs)
            except (ReadError, CompressionError):
                if fileobj is not None:
                    fileobj.seek(saved_pos)
                continue
      raise ReadError("file could not be opened successfully")

E tarfile.ReadError: file could not be opened successfully

anaconda3/envs/oggm_env1/lib/python3.9/tarfile.py:1616: ReadError ------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------- 2022-10-17 17:15:51: oggm.cfg: Reading default parameters from the OGGM params.cfg configuration file. 2022-10-17 17:15:51: oggm.cfg: Multiprocessing switched OFF according to the parameter file. 2022-10-17 17:15:51: oggm.cfg: Multiprocessing: using all available processors (N=4) 2022-10-17 17:15:52: oggm.workflow: init_glacier_directories from prepro level 4 on 1 glaciers. 2022-10-17 17:15:52: oggm.workflow: Execute entity tasks [gdir_from_prepro] on 1 glaciers 2022-10-17 17:15:52: oggm.utils: Downloading https://cluster.klima.uni-bremen.de/~oggm/test_gdirs/oggm_v1.1/RGI61/b_020/L4/RGI60-11/RGI60-11.00.tar to /home/fuxp/OGGM/tests/a3dd618743ed9b57677e1619413c726d/dl_cache/cluster.klima.uni-bremen.de/~oggm/test_gdirs/oggm_v1.1/RGI61/b_020/L4/RGI60-11/RGI60-11.00.tar... 0% of 580.0 KiB | | Elapsed Time: 0:00:00 ETA: --:--:-- 2022-10-17 17:15:54: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_gdirs/oggm_v1.1/RGI61/b_020/L4/RGI60-11/RGI60-11.00.tar 100% of 580.0 KiB |######################| Elapsed Time: 0:00:00 Time: 0:00:00 2022-10-17 17:15:55: oggm.workflow: init_glacier_directories from prepro level 4 on 1 glaciers. 2022-10-17 17:15:55: oggm.workflow: Execute entity tasks [gdir_from_prepro] on 1 glaciers 2022-10-17 17:15:55: oggm.workflow: init_glacier_directories from prepro level 4 on 1 glaciers. 2022-10-17 17:15:55: oggm.workflow: Execute entity tasks [gdir_from_prepro] on 1 glaciers 2022-10-17 17:15:55: oggm.utils: No known hash for cluster.klima.uni-bremen.de/~oggm/test_gdirs/oggm_v1.1/RGI61/b_020/L4/RGI60-11/RGI60-11.00.tar ============================================================================================ warnings summary ============================================================================================= anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/graphics.py:35 anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_surf_to_nan anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_coxe /home/fuxp/anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/graphics.py:35: PendingDeprecationWarning: The get_cmap function will be deprecated in a future version. Use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap(obj) instead. OGGM_CMAPS['section_thickness'] = plt.cm.get_cmap('YlGnBu')

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/graphics.py:37 anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_surf_to_nan anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_coxe /home/fuxp/anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/graphics.py:37: PendingDeprecationWarning: The get_cmap function will be deprecated in a future version. Use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap(obj) instead. OGGM_CMAPS['ice_velocity'] = plt.cm.get_cmap('Reds')

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_benchmarks.py: 4 warnings anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py: 15 warnings anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_models.py: 5 warnings anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py: 69 warnings anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_shop.py: 7 warnings anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_utils.py: 13 warnings /home/fuxp/anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/utils/_workflow.py:2775: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead. for k, s in entity.iteritems():

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_centerlines anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_flowlines anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_downstream anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_width anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_width_corrected anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_graphics.py::test_catch_areas /home/fuxp/anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/graphics.py:55: PendingDeprecationWarning: The get_cmap function will be deprecated in a future version. Use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap(obj) instead. cmap = colormap.get_cmap(cmap, n_colors)

anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py::TestClimate::test_geodetic_mb_calibration_single_fl /home/fuxp/anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py:1918: FutureWarning: The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning. np.testing.assert_allclose(1, mbdf.corr()['new_mb']['old_mb'], atol=0.01)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================================================================= short test summary info ========================================================================================= FAILED anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_prepro.py::TestClimate::test_ref_mb_glaciers - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordin... FAILED anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_shop.py::Test_w5e5::test_process_gswp3_w5e5_data - AssertionError: assert <xarray.DataArray 'year' ()>\narray(1979)\nCoordin... FAILED anaconda3/envs/oggm_env1/lib/python3.9/site-packages/oggm/tests/test_utils.py::TestStartFromOnlinePrepro::test_corrupted_file - tarfile.ReadError: file could not be opened successfully ================================================================== 3 failed, 219 passed, 151 skipped, 126 warnings in 314.47s (0:05:14) ===================================================================

And my OGGM environment are as follows:

OGGM environment:

System info:

python: 3.9.7.final.0
python-bits: 64
OS: Linux
OS-release: 5.15.0-50-generic
machine: x86_64
processor: x86_64

Packages info:

oggm: 1.5.4.dev50+g55651b7
numpy: 1.23.3
scipy: 1.9.1
pandas: 1.5.0
geopandas: 0.11.1
netCDF4: 1.6.0
matplotlib: 3.6.1
rasterio: 1.2.9
fiona: 1.8.20
pyproj: 3.2.1
shapely: 1.8.0
xarray: 2022.9.0
dask: 2022.9.2
salem: 0.3.8
OGGM git identifier: c3defc123252f11b2ac2b6ca68496b00f3c93a0c

Thank you very much!

fmaussion commented 1 year ago

I'm sorry this got lost - if this is still an issue, please re-open