Open RichardScottOZ opened 2 years ago
E.g., tinkering
{'version': 1,
'refs': {'.zgroup': '{"zarr_format":2}',
'SAMPLING_GRID/.zarray': '{\n "chunks": [\n 1\n ],\n "compressor": null,\n "dtype": "<i4",\n "fill_value": 0,\n "filters": null,\n "order": "C",\n "shape": [\n 1\n ],\n "zarr_format": 2\n}',
'SAMPLING_GRID/0': '\x00\x00\x00\x00',
'SAMPLING_GRID/.zattrs': '{\n "_ARRAY_DIMENSIONS": [\n "SAMPLING_GRID"\n ]\n}',
'X/.zarray': '{\n "chunks": [\n 1\n ],\n "compressor": null,\n "dtype": "<f8",\n "fill_value": 0.0,\n "filters": null,\n "order": "C",\n "shape": [\n 0\n ],\n "zarr_format": 2\n}',
'X/.zattrs': '{\n "_ARRAY_DIMENSIONS": [\n "X"\n ]\n}',
'Y/.zarray': '{\n "chunks": [\n 1\n ],\n "compressor": null,\n "dtype": "<f8",\n "fill_value": 0.0,\n "filters": null,\n "order": "C",\n "shape": [\n 0\n ],\n "zarr_format": 2\n}',
'Y/.zattrs': '{\n "_ARRAY_DIMENSIONS": [\n "Y"\n ]\n}',
'Z/.zarray': '{\n "chunks": [\n 1\n ],\n "compressor": null,\n "dtype": "<f8",\n "fill_value": 0.0,\n "filters": null,\n "order": "C",\n "shape": [\n 0\n ],\n "zarr_format": 2\n}',
'Z/.zattrs': '{\n "_ARRAY_DIMENSIONS": [\n "Z"\n ]\n}',
'GEOSCIENCE/Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/.zarray': '{"chunks":[1,938],"compressor":{"id":"zlib","level":9},"dtype":"<f8","fill_value":0.0,"filters":null,"order":"C","shape":[1,15000],"zarr_format":2}',
'GEOSCIENCE/Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/.zattrs': '{"_ARRAY_DIMENSIONS":["SAMPLING_GRID","phony_dim_0"]}',
'GEOSCIENCE/Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/0.Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/0': ['s3://easi-oz/BlockGolem/results/MiraDataPrep/100075.geoh5',
56363221,
3481],
'GEOSCIENCE/Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/0.Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/1': ['s3://easi-oz/BlockGolem/results/MiraDataPrep/100075.geoh5',
56366702,
3289],
'GEOSCIENCE/Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/0.Data/{2e932682-0772-4342-9e13-423af5e8bf80}/Data/2': ['s3://easi-oz/BlockGolem/results/MiraDataPrep/100075.geoh5',
56369991,
3149],
Geoh5py has (N, 3) type things as opposed to (X, Y, Z) as well, which is not so xarray.
So this is exactly the problem. In strict netCDF layout, it's clear which variable depends on which coordinate, and that is what the "_ARRAY_DIMENSIONS"s are about (this is an xarray-specific convention to carry the same information in a zarr dataset).
That the arrays are (N, 3) is not a problem in itself, and I would assume that for every such array, you would get a combined (T, N, 3) where T is the timestamp of each file; but where would these times be stored in the output tree, and is there any tool that can make use of such a zarr dataset? Can Geo5py access multiple HDF files as a single dataset at the moment?
Timestamp or gridid, something like that. I suppose it would be 'top level' as such?
No, no multiple file access - it would be more an analysis tool I think. You can import lots of them into one 'project' as such, but I think this one would be a bit big for a workstation.
The interesting part you wanted 100K histograms for example, without downloading/moving many terabytes, that sort of idea. Or suddenly decided you wanted to look at the Inversion at iteration 5 for that area which was 1876 grid locations.
I have achieved this, too, so thanks!
Would love to have this linked in the kerchunk docs examples page.
Hopefully write something up for you shortly.
@martindurant - here you go https://github.com/RichardScottOZ/Kerchunk-geoh5/blob/main/Kerchunk-Block-Model-Analysis.ipynb
Thank you! That looks very nice and ...complicated :) We'll definitely include a link.
I wonder if you have a simple explanation of what the "z['GEOSCIENCE']['Data']['{00e6c2c7-eefe-4618-b9e6-3fc08177b96b}']['Data']" hierarchy means, and what the (two) array contained therein are.
Also, how do the different input files of total dataset relate to one-another?
Why bananas?!
Yeah, 3D is a bit more complicated.
Yes, I should put a link in at least to the geoh5py docs
Banana Splits, first thing I thought of that is obviously not a real work bucket!
An example of an hdf5 file format.
https://geoh5py.readthedocs.io/en/stable/
"The geoh5py library has been created for the manipulation and storage of a wide range of geoscientific data (points, curve, surface, 2D and 3D grids) in geoh5 file format. Users will be able to directly leverage the powerful visualization capabilities of Geoscience ANALYST along with open-source code from the Python ecosystem."
So I use it quite a bit.
In fact, now have 200K similar files, in a couple of company s3 buckets.
e.g.![image](https://user-images.githubusercontent.com/72196131/160500793-aaa1d882-f999-4723-ba4e-246d76ad3567.png)
So last night I started wondering if could access them quickly from s3 as opposed to generally read each one, get what I want and have several terabytes lying around.
I remembered the pangeo netcdf/hdf5 discussion etc, so I started having a look.
This would be pretty useful.
e.g. something like this? @rsignell-usgs