Running TopoPyScale examples on Mac (M3)

mvantiggelen commented 3 months ago

Hi all

When running the TopoPyScale examples on Mac (ARM M3) I get some IO errors with h5py, probably due to the fact that the tmp files are opened at the same time and then deleted by different processes. When running for 'CPU_cores: 1 ' I still get a similar error. The code endlessly restarts by itself but never really crashes. The same code runs fine on a Linux machine.

Any ideas ? Have there been any other tests for Mac ? I'm not 100% sure this is Mac related.

Below is the output when running pipeline_test.py from ex1_norway_finse (I've attached the config file(config_test.txt), the pipeline file (pipeline_test.txt and the list of the conda packages I'm using (conda_env.txt))

Best,

Maurice van Tiggelen

 ------------------------------- 

Toposub Spatial Downscaling Example

 ------------------------------- 

---> Output directory cleaned
---> no FSM directory to clean
---> no ensemble directory to clean

---> DEM file found
{'latN': 60.68443065002782, 'latS': 60.50790355399738, 'lonW': 7.354447266990793, 'lonE': 7.731798660953054}
Project lat/lon extent:

    ------------------------------
    |        North:60.7          |
    |West:7.4          East:7.7|
    |        South:60.5          |
    ------------------------------

---> Loading ERA5 surf climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF SURF data found:
0    SURF_201810.nc
1    SURF_201811.nc
2    SURF_201812.nc
Name: target_file, dtype: object

---> Loading ERA5 plev climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF PLEV data found:
0    PLEV_201810.nc
1    PLEV_201811.nc
2    PLEV_201812.nc
Name: target_file, dtype: object

---> No ds_param.nc Dataset found. DEM /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse//inputs/dem/ASTER_Finse.tif available.

---> Extracting DEM parameters (slope, aspect, svf)
Computing slope and aspect ...
Computing svf ...
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
---> Scaling data prior to clustering
---> Clustering with Mini-Batch K-means in 5 clusters
---> Mini-Batch Kmean finished in 0.0s
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
Convert coordinates from EPSG:32632 to EPSG:4326
---> Centroids file df_centroids.pck saved

---> Computing solar geometry
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_solar.nc saved

---> Computing horizons with 10 degree increments
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/da_horizon.nc saved
---> Centroids file df_centroids.pck updated with horizons

-------------------------------------------------
           TopoScale - Downscaling

---> Downscaling climate to list of points using TopoScale
/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/tmp cleaned
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing plev for point 0
Preparing plev for point 1
Preparing plev for point 2
Preparing plev for point 3
Preparing plev for point 4
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing surf for point 0
Preparing surf for point 1
Preparing surf for point 2
Preparing surf for point 3
Preparing surf for point 4

 ------------------------------- 

 ------------------------------- 

Toposub Spatial Downscaling Example
Toposub Spatial Downscaling Example

 ------------------------------- 

 ------------------------------- 

Traceback (most recent call last):
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/TopoPyScale/topoclass.py", line 68, in __init__
    shutil.rmtree(self.config.outputs.path)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/shutil.py", line 734, in rmtree
---> Output directory cleaned
---> no FSM directory to clean
---> no ensemble directory to clean
    _rmtree_safe_fd(fd, path, onerror)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())

---> DEM file found  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd

    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'ds_param.nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 288, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/pipeline_test.py", line 9, in <module>
    mp = tc.Topoclass(config_file)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/TopoPyScale/topoclass.py", line 71, in __init__
    os.makedirs(self.config.outputs.path)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs'
{'latN': 60.68443065002782, 'latS': 60.50790355399738, 'lonW': 7.354447266990793, 'lonE': 7.731798660953054}
Project lat/lon extent:

    ------------------------------
    |        North:60.7          |
    |West:7.4          East:7.7|
    |        South:60.5          |
    ------------------------------

---> Loading ERA5 surf climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF SURF data found:
0    SURF_201810.nc
1    SURF_201811.nc
2    SURF_201812.nc
Name: target_file, dtype: object

---> Loading ERA5 plev climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF PLEV data found:
0    PLEV_201810.nc
1    PLEV_201811.nc
2    PLEV_201812.nc
Name: target_file, dtype: object

---> No ds_param.nc Dataset found. DEM /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse//inputs/dem/ASTER_Finse.tif available.

---> Extracting DEM parameters (slope, aspect, svf)
Computing slope and aspect ...
Computing svf ...
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
---> Scaling data prior to clustering
---> Clustering with Mini-Batch K-means in 5 clusters
---> Mini-Batch Kmean finished in 0.0s
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
Convert coordinates from EPSG:32632 to EPSG:4326
---> Centroids file df_centroids.pck saved

---> Computing solar geometry
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_solar.nc saved

---> Computing horizons with 10 degree increments
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/da_horizon.nc saved
---> Centroids file df_centroids.pck updated with horizons

-------------------------------------------------
           TopoScale - Downscaling

---> Downscaling climate to list of points using TopoScale
/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/tmp cleaned
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing plev for point 0
Preparing plev for point 1

 ------------------------------- 

Toposub Spatial Downscaling Example

 ------------------------------- 

---> Output directory cleaned
---> no FSM directory to clean
---> no ensemble directory to clean

---> DEM file found
{'latN': 60.68443065002782, 'latS': 60.50790355399738, 'lonW': 7.354447266990793, 'lonE': 7.731798660953054}
Project lat/lon extent:

    ------------------------------
    |        North:60.7          |
    |West:7.4          East:7.7|
    |        South:60.5          |
    ------------------------------

---> Loading ERA5 surf climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF SURF data found:
0    SURF_201810.nc
1    SURF_201811.nc
2    SURF_201812.nc
Name: target_file, dtype: object

---> Loading ERA5 plev climate forcing
Start =  2018-Oct
End =  2018-Dec
ECWMF PLEV data found:
0    PLEV_201810.nc
1    PLEV_201811.nc
2    PLEV_201812.nc
Name: target_file, dtype: object

---> No ds_param.nc Dataset found. DEM /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse//inputs/dem/ASTER_Finse.tif available.

---> Extracting DEM parameters (slope, aspect, svf)
Computing slope and aspect ...
Computing svf ...
Preparing plev for point 2
Preparing plev for point 3
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
---> Scaling data prior to clustering
---> Clustering with Mini-Batch K-means in 5 clusters
---> Mini-Batch Kmean finished in 0.0s
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_param.nc saved
Convert coordinates from EPSG:32632 to EPSG:4326
---> Centroids file df_centroids.pck saved

---> Computing solar geometry
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/ds_solar.nc saved

---> Computing horizons with 10 degree increments
Preparing plev for point 4
---> File /Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/da_horizon.nc saved
---> Centroids file df_centroids.pck updated with horizons

-------------------------------------------------
           TopoScale - Downscaling

---> Downscaling climate to list of points using TopoScale
/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/tmp cleaned
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing plev for point 0
Preparing plev for point 1
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing surf for point 0
Preparing surf for point 1
Preparing surf for point 2
Preparing surf for point 3
Preparing surf for point 4
Preparing plev for point 2
Preparing plev for point 3
Traceback (most recent call last):
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
KeyError: [<class 'h5netcdf.core.File'>, ('/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/outputs/tmp/ds_plev_pt_2.nc',), 'r', (('decode_vlen_strings', True), ('invalid_netcdf', None)), '3c6122fe-2263-432f-bd1c-5129e91f4c7c']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 288, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/Tigge006/surfdrive/04_Scripts/Python/TopoPyScale_examples/ex1_norway_finse/pipeline_test.py", line 15, in <module>
    mp.downscale_climate()
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/TopoPyScale/topoclass.py", line 627, in downscale_climate
    ta.downscale_climate(self.config.project.directory,
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/TopoPyScale/topo_scale.py", line 477, in downscale_climate
    plev_pt_list.append(xr.open_dataset(output_directory / f'tmp/ds_plev_pt_{row.point_name}.nc', engine='h5netcdf'))
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/api.py", line 525, in open_dataset
    backend_ds = backend.open_dataset(
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 413, in open_dataset
    store = H5NetCDFStore.open(
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 176, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 127, in __init__
    self._filename = find_root_and_group(self.ds)[0].filename
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 187, in ds
    return self._acquire()
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/h5netcdf_.py", line 179, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/h5netcdf/core.py", line 1051, in __init__
    self._h5file = self._h5py.File(
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/Users/Tigge006/opt/anaconda3/envs/downscaling/lib/python3.9/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
BlockingIOError: [Errno 35] Unable to synchronously open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
Preparing plev for point 4
No ERA5T  PRESSURE data present with additional dimension <expver>
Preparing surf for point 0
Preparing surf for point 1
Preparing surf for point 2
Preparing surf for point 3
Preparing surf for point 4

 ------------------------------- 

Toposub Spatial Downscaling Example

 ------------------------------- 

---> Output directory cleaned
---> no FSM directory to clean
---> no ensemble directory to clean

---> DEM file found
{'latN': 60.68443065002782, 'latS': 60.50790355399738, 'lonW': 7.354447266990793, 'lonE': 7.731798660953054}
Project lat/lon extent:

    ------------------------------
    |        North:60.7          |
    |West:7.4          East:7.7|
    |        South:60.5          |
    ------------------------------

---> Loading ERA5 surf climate forcing

.
.
.
.
.
.

ArcticSnow commented 3 months ago

Hello @mvantiggelen ,

It was installed succesfully on a Mac, but maybe not an ARM architecture one. I have never tried myself. It is acutally Varya (@varyabazilova ), close to you that installed it on a Mac. So you may try to have a chat with her.

It may have to do how the Python multiprocessing toolbox is running on ARM processor. If you find a fix let us know

ArcticSnow commented 3 months ago

Any success @mvantiggelen ?

mvantiggelen commented 3 months ago

Any success @mvantiggelen ?

No solution yet! I've temporally switched to a Linux cluster for testing

kyung-robin-kim commented 3 months ago

Hello, I've also been running into the same problem on Mac, I'll try switching over to Linux as well for the time being.... I don't know if it's related, but Andrew's comment about topocalc not working with the Mac environment makes me wonder.

kyung-robin-kim commented 2 months ago

I have a question/request. Could I request/schedule a tutorial at some point? I had to take a break from trying to get TopoPyScale to run on my computer, and would like some additional help if possible, time willing...

ArcticSnow / TopoPyScale

Running TopoPyScale examples on Mac (M3) #109