alan-turing-institute / clim-recal

Open repository of methods for recalibrating & bias correcting UKCP18 climate projections data
https://alan-turing-institute.github.io/clim-recal/
MIT License
10 stars 1 forks source link

Segmentation fault converting `CPM` to 365/366 day calendars/`277000` coordinates #149

Closed griff-rees closed 5 months ago

griff-rees commented 5 months ago

Segmentation fault after completing 110 conversions from raw CPM.nc files:

...
  99% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  99/100  [ 1:07:22 < -:--:-- , ? it/s ]
Warning 1: Latitude grid not spaced evenly.  Setting projection for grid spacing is
within 0.1 degrees threshold.
 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100/100  [ 1:07:34 < 0:00:00 , ? it/s ]
<CPMResampler(count=100, max_count=100,
input_path='/mnt/vmfileshare/ClimateData/Raw/UKCP2.2/tasmax/06/latest',
output_path='/mnt/vmfileshare/ClimateData/CPM-365/tif-projection-rewrite/sixth-run/run_24-05-21_02-57/resample/cpm/tasmax/06/latest')>
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_latitude_bnds']
  simplified_netcdf: Dataset = open_dataset(
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_longitude_bnds']
  simplified_netcdf: Dataset = open_dataset(
   0%                                          0/100  [ 0:00:28 < -:--:-- , ? it/s 
]Warning 1: Latitude grid not spaced evenly.  Setting projection for grid spacing is
within 0.1 degrees threshold.
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_latitude_bnds']
  simplified_netcdf: Dataset = open_dataset(
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_longitude_bnds']
  simplified_netcdf: Dataset = open_dataset(
   1% ━                                       1/100  [ 0:01:07 < -:--:-- , ? it/s ]
...
   9% ━━━                                     9/100  [ 0:06:24 < -:--:-- , ? it/s ]
Warning 1: Latitude grid not spaced evenly.  Setting projection for grid spacing is within 0.1 degrees threshold.
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_latitude_bnds']  simplified_netcdf: Dataset = open_dataset(
/home/grees/code/clim-recal/python/clim_recal/utils/xarray.py:334: UserWarning:
Variable(s) referenced in bounds not in variables: ['grid_longitude_bnds']
  simplified_netcdf: Dataset = open_dataset(

  10% ━━━                                    10/100  [ 0:07:04 < -:--:-- , ? it/s ]
Warning 1: Latitude grid not spaced evenly.  Setting projection for grid spacing is
within 0.1 degrees threshold.
  10% ━━━                                    10/100  [ 0:07:17 < -:--:-- , ? it/s ]
There are 1247 HDF5 objects open!

Report: open objects on 72057594037928835
Type = File(72057594037927936) name='/'
Type = File(72057594037927937) name='/'
Type = File(72057594037927938) name='/'
Type = File(72057594037927939) name='/'
Type = File(72057594037927940) name='/'
Type = File(72057594037927941) name='/'
Type = File(72057594037927942) name='/'
Type = File(72057594037927943) name='/'
Type = File(72057594037927944) name='/'
Type = File(72057594037927945) name='/'
Type = File(72057594037927946) name='/'
Type = File(72057594037927947) name='/'
Type = File(72057594037928796) name='/'
Type = File(72057594037928804) name='/'
Type = File(72057594037928812) name='/'
Type = File(72057594037928820) name='/'
Type = File(72057594037928828) name='/'
Type = File(72057594037928834) name='/'
Segmentation fault (core dumped)
griff-rees commented 5 months ago

This is actually a case of no more hard drive space. Closing

griff-rees commented 5 months ago

Another example for future reference:

There are 493 HDF5 objects open!

Report: open objects on 72057594037928690
Type = File(72057594037927936) name='/'Type = File(72057594037928673) name='/'Type
= File(72057594037928681) name='/'Type = File(72057594037928689) name='/'Segmentation fault (core dumped)