Open Thomas-Moore-Creative opened 9 months ago
There is detail and nuance in this choice but for this task we'll for now define:
Using ONI:
ONI (5N-5S, 170W-120W): The ONI uses the same region as the Niño 3.4 index. The ONI uses a 3-month running mean, and to be classified as a full-fledged El Niño or La Niña, the anomalies must exceed +0.5C or -0.5C for at least five consecutive months. This is the operational definition used by NOAA.
References:
Repos: make NCAR's ONI index analysis ready locally #1
NCAR
Nino SST Indices (Nino 1+2, 3, 3.4, 4; ONI and TNI)
NOAA Climate explainers:
El Niño Southern Oscillation (ENSO)
Multivariate ENSO Index Version 2 (MEI.v2)
Why are there so many ENSO indexes, instead of just one?
- You cannot blame EVERYTHING on El Niño.
- No two El Niños are alike, and the same goes for their impacts.
- And, Just because a strong event is forecasted does not necessarily mean strong impacts. Instead, El Niño can increase the chances for certain types of extremes.
Conversation: El Niño combined with global warming means big changes for New Zealand’s weather
What North America can expect from El Niño
Peer-reviewed publications:
Observing and Predicting the 2015/16 El Niño
The Role of Buoy and Argo Observations in Two SST Analyses in the Global and Tropical Pacific Oceans
Todo:
Starting basic composites - means over all event-months.
El Nino = 8 events ( weak - strong ) as defined by ONI La Nina = 10 events ( weak - strong ) as defined by ONI El Nino months total = 70 = 19% La Nina months total = 119 = 33% Neutral months total = 171 = 48%
Under 8 minutes for the write of the 3 climatologies
Thanks to an idea from @ChrisC28 I'm going to compare the performance of using .select([list of times])
vs .where(Bool_mask)
for the compositing masking.
A single quantile calculation using:
El_Nino_mask_TIMES = El_Nino_mask01['Time'].where(El_Nino_mask,drop=True)
result = temp_chunked_time.sel({'Time':El_Nino_mask_TIMES.values}).quantile([0.05],skipna=True,dim='Time').compute()
takes -
CPU times: user 12min 9s, sys: 1min 43s, total: 13min 53s
Wall time: 2h 18min 6s
.quantile([0.05],skipna=**False**,dim='Time')
fixed the unresponsive calculation. Apparently numpy
.nanquantile
is 100x slow?!?
Also using .sel
on a list of masked and dropped times is 2x faster than pure .where
30GB "stats" ncfile writing out with a rechunk very effectively . . . max memory at about 50GB
6.5 minutes for climatologies and 21.5 minutes for stats for a single variable
If we plan on "less than 45 minutes" per batched variable then total time, once resources are available, should be less than 5 hours.
How can I ensure I control chunk sizes in the output NetCDF
files?
ds.to_netcdf("/tmp/test1.nc", encoding={"z": {"chunksizes": (5, 5)}})
encoding?
BRAN2020_temperature_climatology.to_netcdf(write_path+'BRAN2020_temperature_climatology.nc',format='netCDF4')
yields an NC file with "contiguous" storage:
float La_Nina_climatological_temp(month, st_ocean, yt_ocean, xt_ocean) ;
La_Nina_climatological_temp:_FillValue = NaNf ;
La_Nina_climatological_temp:cell_methods = "time: mean Time: mean" ;
La_Nina_climatological_temp:coordinates = "geolon_t geolat_t" ;
La_Nina_climatological_temp:long_name = "Potential temperature" ;
La_Nina_climatological_temp:packing = 4LL ;
La_Nina_climatological_temp:standard_name = "sea_water_potential_temperature" ;
La_Nina_climatological_temp:time_avg_info = "average_T1,average_T2,average_DT" ;
La_Nina_climatological_temp:units = "degrees C" ;
La_Nina_climatological_temp:valid_range = -32767LL, 32767LL ;
La_Nina_climatological_temp:_Storage = "contiguous" ;
La_Nina_climatological_temp:_Endianness = "little" ;
NetCDF
settings = dict(chunksizes={'month':1,'st_ocean':10, 'yt_ocean':1500, 'xt_ocean':3600})
encoding = {var: settings for var in BRAN2020_temperature_climatology.data_vars}
BRAN2020_temperature_climatology.to_netcdf(write_path+'BRAN2020_temperature_climatology.nc',encoding = encoding)
is yielding error:
File /g/data/v14/tm4888/miniconda3/envs/busecke_etal_grl_2019_omz_euc/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:495, in NetCDF4DataStore.prepare_variable(self, name, variable, check_encoding, unlimited_dims)
493 nc4_var = self.ds.variables[name]
494 else:
--> 495 nc4_var = self.ds.createVariable(
496 varname=name,
497 datatype=datatype,
498 dimensions=variable.dims,
499 zlib=encoding.get("zlib", False),
500 complevel=encoding.get("complevel", 4),
501 shuffle=encoding.get("shuffle", True),
502 fletcher32=encoding.get("fletcher32", False),
503 contiguous=encoding.get("contiguous", False),
504 chunksizes=encoding.get("chunksizes"),
505 endian="native",
506 least_significant_digit=encoding.get("least_significant_digit"),
507 fill_value=fill_value,
508 )
510 nc4_var.setncatts(attrs)
512 target = NetCDF4ArrayWrapper(name, self)
File src/netCDF4/_netCDF4.pyx:2838, in netCDF4._netCDF4.Dataset.createVariable()
File src/netCDF4/_netCDF4.pyx:4085, in netCDF4._netCDF4.Variable.__init__()
KeyError: 0
where: encoding =
{'climatological_temp': {'chunksizes': {'month': 1,
'st_ocean': 10,
'yt_ocean': 1500,
'xt_ocean': 3600}},
'El_Nino_climatological_temp': {'chunksizes': {'month': 1,
'st_ocean': 10,
'yt_ocean': 1500,
'xt_ocean': 3600}},
'La_Nina_climatological_temp': {'chunksizes': {'month': 1,
'st_ocean': 10,
'yt_ocean': 1500,
'xt_ocean': 3600}}}
daily
datadaily
rechunking operationzarr
collections using single node mega memory resources with 800GB JobFSclimatology
and quantile
calculations on chunked4time
collection of daily temperature data, both running in under 70 minutes using XXLMEM on the normalbw queue.Tests using 3D daily temp
variable show that for all 6 variables:
nci-file-expiry
could mean re-running ARD might be quarterly or yearly?for some reason 'salt' failed due to being killed because of JobFS? But why is the quota 400 if I requested 800?
Job 110281744.gadi-pbs killed due to exceeding jobfs quota. Quota: 400.0GB, Used: 418.91GB, Host: gadi-mmem-clx-0001 /g/data/es60/users/thomas_moore/miniconda3/envs/pangeo_regionmask/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 50 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '
======================================================================================
Resource Usage on 2024-03-08 01:31:14:
Job Id: 110281744.gadi-pbs
Project: es60
Exit Status: 271 (Linux Signal 15 SIGTERM Termination)
Service Units: 2288.53
NCPUs Requested: 96 NCPUs Used: 96
CPU Time Used: 85:05:00
Memory Requested: 5.86TB Memory Used: 2.66TB
Walltime requested: 10:00:00 Walltime Used: 04:46:04
JobFS requested: 800.0GB JobFS used: 418.91GB
======================================================================================
Job 111169419.gadi-pbs killed due to exceeding jobfs quota. Quota: 1.37TB, Used: 1.38TB, Host: gadi-mmem-clx-0002
======================================================================================
Resource Usage on 2024-03-18 22:25:36:
Job Id: 111169419.gadi-pbs
Project: es60
Exit Status: 271 (Linux Signal 15 SIGTERM Termination)
Service Units: 1233.33
NCPUs Requested: 96 NCPUs Used: 96
CPU Time Used: 50:36:38
Memory Requested: 5.84TB Memory Used: 2.71TB
Walltime requested: 24:00:00 Walltime Used: 02:34:10
JobFS requested: 2.73TB JobFS used: 1.38TB
======================================================================================
intermittent networking issues between nodes appears to cause challenges with two nodes? ???
======================================================================================
Resource Usage on 2024-03-20 12:39:53:
Job Id: 111301310.gadi-pbs
Project: es60
Exit Status: 271 (Linux Signal 15 SIGTERM Termination)
Service Units: 582.47
NCPUs Requested: 48 NCPUs Used: 48
CPU Time Used: 48:11:05
Memory Requested: 2.92TB Memory Used: 2.67TB
Walltime requested: 24:00:00 Walltime Used: 02:25:37
JobFS requested: 1.37TB JobFS used: 1.38TB
======================================================================================
Monthly climatologies for:
Over BRAN2020 variables:
Generating these statistics:
workflow:
delivery