Onboard wind hazard data

[joemoorhouse]

Open-ended issue to look at sources for wind data suitable for residential real estate. e.g. https://climate.copernicus.eu/vortex

[joemoorhouse]

Hi @DavideFerri, @floriangallo, @aliebadi22, @dbferri,

I have been looking at https://www.ngfs.net/ngfs-scenarios-portal/explore and in particular http://climate-impact-explorer.climateanalytics.org/methodology/ 2.3.3. Tropical Cyclone 'The tropical cyclone modelling consists of two steps: first, generating a probabilistic track set from historical tracks, and second, computing the wind fields at centroid points and performing the climate change scaling.' This might be a good way to tackle wind, both tropical and potentially ex-tropical. I think we may be looking at running someone else's model to create a data set rather than use a data set directly.

CLIMADA is in many ways a similar project to OS-C albeit with more emphasis on the hazard event modelling and a wider focus (interaction between OS-C and CLIMADA a topic for another day!)

The CLIMADA approach looks interesting https://climada-python.readthedocs.io/en/stable/tutorial/climada_hazard_TropCyclone.html and I think the model/code is available including 'climate change scaling'

I am wondering if we, OS-C, should be running their model to create our stored-down dataset?; I think assumption here is that we will not be simulating on-the-fly at scale.

[joemoorhouse]

++ @aliebadi22

[DavideFerri]

Hi @joemoorhouse

I had a look at Vortex and, in particular, at this: https://globalwindatlas.info/. I understand it is meant to inform on what returns a wind farm would generate, but have you had a look at the underlying data to see if we could also use it?

Re https://climada-python.readthedocs.io/en/stable/tutorial/climada_hazard_TropCyclone.html, I also find very interesting what they have. What data were you thinking to capture?

[joemoorhouse]

The CGFI team (see also https://www.cgfi.ac.uk/global-resilience-index-initiative/try-grii-now/) pointed us to this set by Bloemendaal et al (2020): STORM tropical cyclone wind speed return periods.

https://data.4tu.nl/articles/dataset/STORM_climate_change_tropical_cyclone_wind_speed_return_periods/14510817/3 and then https://figshare.com/s/397aff8631a7da2843fc

This indeed looks very useful and I suggest we prioritize.

From a very quick look, we probably want to take data and convert to a single global zarr file with WGS 84 CRS.

[joemoorhouse]

Hi @mvarfima, Is this one you (or colleague) might like to take on? Bit of work there (probably in hazard repo) to do the necessary conversion to OS-C zarr format. I have found it works well to convert first to xarray; easy to save to our zarr format from there.

[mvarfima]

Hi @joemoorhouse.

I was inspecting both data sources and I think I need some guidance.

1. For the STORM climate change tropical cyclone wind speed return periods datasets I think we are interested in the "STORM_FIXED_RETURN_PERIODS_CMCC" folders for every .nc file inside (which cover different regions). I can converts from (0, 360)º to (-180, 180) using ecmwf conversor and create transfrom for declarated 10km resolution.
2. For the STORM Climate Change synthetic tropical cyclone tracks we have reanalysis data for 4 different models. That data is used to produce future climate conditions with storm(see paper and scenario data)

So, the datasets from point 2 are used to obtein RCP8.5 future data, point 1.

If you find it apropiate I can create a notebook to upload reanalysis data (4 models) + scenario data using declared coordnate system and resolution, and onboard it to s3 in zarr type as I did with JRC. We could comment the results in the next OS-C Wednesay session.

[joemoorhouse]

Hi @mvarfima,

For 1, in the hazard repo we are tending to onboard both .nc and .tif straight into xarrays and then perform any necessary projections from there. See for an example of a re-projection https://github.com/os-climate/hazard/blob/main/src/hazard/utilities/map_utilities.py transform_epsg4326_to_epsg3857 I think fine to put something as a notebook to start with to get it working. Then we would probably turn it into a separate class similar to things like: https://github.com/os-climate/hazard/blob/main/src/hazard/models/work_loss.py with a new 'source' If you stick with xarrays that will take writing the classes easy later. e.g. we have https://github.com/os-climate/hazard/blob/main/src/hazard/sources/osc_zarr.py Which can write xarrays to zarr in the correct conventions. So actually, the work is really to get the data into a single xarray with the correct CRS (probably WGS 84) and bounds.

2 sounds interesting, but perhaps we split into two issues: this one for the hazard map and another for the events?

[mvarfima]

Hi @joemoorhouse.

Sorry for the late response and thank you for the links, it's good to have a reference.

1. Okey, I didn't see the tif files till now. So, do you have any code related to the onborading of this data in the repo? I was thinking that i can create the code to unzip every folder and onboard every .nc and .tif file. The destination folder can have the same names and structures as the source one. I would say that this task can be done straigtforward if the resolution and affine transformation of the maps are the same. If not, we can recover the CRS and use pyproj library.

2. So then we can treat this events datasource in a different thread and keep this one for maps. I suggest following the same methodology that in 1. to onboard this data.

So, my code structure proposal covers:

1. Unzipping.
2. Finding every .nc and .tif file.

For every hazard map:

4. Creating destination path in s3 (recovered from source folder and file names. Use s3fs and/or boto3 when needed.
5. Recover affine transformation and shape. Fix resolution to 10km. Transfrom to Lat-Lot using pyproj or osc utilities if exist.
6. Create zarr array with given shape and transform. Use oscZarr() method.
7. Populate zarr array. Use oscZarr() method.

Let me know what you think and if there is any other osc utility tht I should be aware of. I can deliver a solution if you lke the idea.

[NickKellett]

This may be helpful: "OGC Testbed 17: COG/Zarr Evaluation Engineering Report" https://docs.ogc.org/per/21-032.html

[joemoorhouse]

Hi @mvarfima,

Holidays got in the way, but I tried out a few ways to combine multiple small arrays into one large chunked array. I have an approach I am pretty happy with, but still in a branch for now as not quite complete. https://github.com/joemoorhouse/hazard/tree/inventory-test

I added a test that is more like a script to download and run the conversion routine: test_wind_onboarding. The main code is here: https://github.com/joemoorhouse/hazard/blob/inventory-test/src/hazard/onboard/storm_wind.py The logic to combine arrays is in the method add_children_to_parent https://github.com/joemoorhouse/hazard/blob/inventory-test/src/hazard/utilities/xarray_utilities.py

You'll see that I load the 'children' to be combined as xarrays (DataArrays in fact); it is a useful intermediate storage. We can use the xarray coordinates or Affine transforms to work out the parent's indices that correspond to the child. I show both ways. I keep the target as a zarr array as I find it easier to deal with that directly.

Thanks Joe

[joemoorhouse]

Hi @obsidjan, @mvarfima, @floriangallo As an update on wind hazard models, the remaining work in this issue is to complete the on-boarding into OS-C of the STORM tropical cyclone hazard maps: https://data.4tu.nl/articles/dataset/STORM_climate_change_tropical_cyclone_wind_speed_return_periods/14510817/3 https://figshare.com/s/397aff8631a7da2843fc This on-boarding is implementing in the hazard.

Separately we are discussing on-boarding of IRIS both as hazard map and (synthetic) event set; I will create a separate issue for that (most likely in hazard directly).