Add ESGF ini file or equivalent information

matthew-mizielinski commented 2 years ago

ESGF ini file for CMIP6:

https://github.com/ESGF/config/blob/master/publisher-configs/ini/esg.cmip6.ini

Much of this can be derived from the mip-cmor-tables and the fields to be added to this repo

matthew-mizielinski commented 2 years ago

Can we separate this into a generic and a project specific component?

durack1 commented 2 years ago

Totally, I figure we can harvest info from the MIP-CMOR-tables along with the project info, and during a reconfiguration of ESGF dupe fields can be dropped. @sashakames is that what you were thinking?

matthew-mizielinski commented 2 years ago

Totally, I figure we can harvest info from the MIP-CMOR-tables along with the project info, and during a reconfiguration of ESGF dupe fields can be dropped. @sashakames is that what you were thinking?

Have just invited sasha to this repo

durack1 commented 9 months ago

@sashakames @wolfiex it would be great to get both the ESGF/COG ini and metagrid configs sorted ASAP - is this week pushing out luck Sasha?

sashakames commented 9 months ago

esg.cmip6.ini is used by esgmapfile, still used by legacy esg-publisher installations, (but I will not support those, they need to be upgraded.) This file is automatically generated by the script, derived directly from the CV I had added CMIP6Plus to Metagrid, perhaps there is a PR somewhere...
For COG, this is a frontend change, why don't we add a link to Metagrid to get people away from the site we plan to decommission soon.

durack1 commented 9 months ago

@sashakames so I found these COG related files: https://github.com/ESGF/config/blob/master/search-configs/search_cmip6.cfg https://github.com/ESGF/config/blob/master/publisher-configs/ini/esg.cmip6.ini (old publisher, what is the new one?)

Did we need other configs for metagrid, or does it read the same format?

It make sense to me that moving forward these things would be json, if that format makes sense to the dev-side

sashakames commented 9 months ago

If Metagrid does not have a test suite ready soon after we are ready, and sites are still balking on upgrading without that (though its been running fine at LLNL for a long time), sure I could help with the CoG config. However, I think we are very close now, and then will be reasonable to require sites to upgrade for CMIP6Plus support. Does CEDA have a timeframe to start publishing yet?

ESGF already officially deprecated the old software stack so we are under so obligation to support COG.

Metagrid doesn't have individual files, neither does the new publisher, as its based on CMIP6 as a "stock" project.

sashakames commented 9 months ago

https://esgf-dev1.llnl.gov/search?project=CMIP6Plus Unless I wipe the db, this should stay persistent for demo purposes. We can reach out to Carlos for a date to update the main beta server and Nimbus will follow.

durack1 commented 7 months ago

Having a look at the esg.cmip6.ini, the following fields would need a tweak, whereas other fields could likely be directly used from the CMIP6 template"

[project:cmip6]

activity_drs_options = AerChemMIP, C4MIP, CDRMIP,

categories = 
    mip_era            | enum   | true  | true  | 0 
    activity_drs       | enum   | true  | true  | 1 
    institution_id     | enum   | true  | true  | 2
    ...

category_defaults = project | CMIP6

experiment_id_options = 1pctCO2, 1pctCO2-4xext, ...

experiment_title_map = map(experiment_id : experiment_title)
    1pctCO2                     | 1 percent per year increase in CO2                                                                                                                                           
    1pctCO2-4xext               | extension from year 140 of 1pctCO2 with 4xCO2                                                                                                                                
    1pctCO2-bgc                 | biogeochemically-coupled version of 1 percent per year increasing CO2 experiment
    ...

institution_id_options = AER, AS-RCEC, AWI, ...

min_cmor_version = 3.2.4

min_cf_version = 1.6

min_data_specs_version = 01.00.13

mip_era_options = CMIP6

model_cohort_map = map(source_id : model_cohort)
    4AOP-v1-5                 | Published 
    ACCESS-CM2                | Published 
    ACCESS-ESM1-5             | Published
    ...

project_options = CMIP6

source_id_options = 4AOP-v1-5, ACCESS-CM2, ACCESS-ESM1-5, ...

sub_experiment_id_options = none, s1910, s1920, ...

sashakames commented 7 months ago

AFAIK there has not been an update to esgprep/esgf-prepare the module that uses the .ini file, they are still stuck on python2. So I'd say we can pass on updating that for now.

durack1 commented 7 months ago

@sashakames this is required for the COG interfaces at DKRZ and CEDA, to configure the CMIP6Plus project, no?

sashakames commented 7 months ago

No, the file is not used for CoG. There is an optional input file for CoG project definitions and facets but its fairly trivial to generate. They can copy the CMIP6 and drop any facets we plan to omit, eg. product .
But more broadly speaking, why support CoG at all when Metagrid is ready, otherwise they will waste time with a deprecated platform. I know CEDA is close now to having a working Metagrid, DKRZ is up to Katharina who is indeed time constrained.

durack1 commented 7 months ago

@sashakames great, we had queries about how close CEDA is the Metagrid deployment, if there are close, excellent. If DKRZ is a while off, I wonder how we help that process along?

sashakames commented 7 months ago

Its a resource issue, of course we can try to simply things as much as possible, but sounds like she has to do 3 jobs or something... She has been maintaining CoG and could probably make the call which suits her. Other CoG sites are IPSL, LIU and NCI, we can check on their plans. Next CDNOT meeting not till 14 Feb, lets see where we are at if earlier engagement is needed.

WCRP-CMIP / CMIP6Plus_CVs

Add ESGF ini file or equivalent information #3