jbusecke / esgf-virtual-zarr-data-access

ESGF working group to enable data access via virtual zarrs.
Apache License 2.0
5 stars 1 forks source link

Performance of large reference files #3

Open jbusecke opened 2 months ago

jbusecke commented 2 months ago

We should explore what the 'worst case' scenario of a file for CMIP7 is in terms of creating a very large reference file (which might require us to write the data to a parquet file).

It would be really valuable to get estimates of the output size from modeling centers? E.g. what is the highest time/space resolution output that will be produced?