Solar data tiny tweak - targeting SOLARIS-HEPPA-CMIP-4-4

[znichollscr]

@st-bender @berndfunke 1000 apologies for this, I should have caught it earlier. For your next version (whenever that is), there is one tiny other tweak to your solar data. Can you please remove the attributes datetime_end and datetime_start global attributes in all your files. They're not written correctly at the moment (the format is meant to depend on the file's frequency, a bit of a headache) and we can actually just infer them from the time axis in your data, so we can avoid sparing you the details of the frequency-dependent formatting.

Apologies again, I know we asked you to put that there in the first place. For the 4.3 data, it doesn't matter, just leave it as is.

One more thing for the list: we should double check the creation_date attribute. This should be picked up by the new version of input4mips-validation, but just in case, here is a reminder. If the solar team want to write this themselves, it must be the timestamp when the file was created (in the UTC timezone), in the format "YYYY-mm-ddTHH:MM:SSZ" (the ISO8601 standard).

Thanks and sorry again!

cc @durack1

[durack1]

@znichollscr, we haven't published 4.3, so we could catch this before we do. Actually, I see that 4.3 is available from https://solarisheppa.geomar.de/cmip7 so we'd need to increment again to 4.4.. This dual data download sources is causing some challenges..

@berndfunke as we have this ESGF publication process starting to happen quickly, would you be happy if we publish data in one place only and then link to this from the SOLARIS-HEPPA website? It seems having these data in two places is going to complicate our communication a lot

As an example, on the SOLARIS-HEPPA website you could have a link like this which means users get the same data from a single download source?

[berndfunke]

Hi @durack1, this should work but it would be good to link the files individually. Is this possible?

El 3 ago 2024, a las 0:01, Paul J. Durack @.***> escribió:

@znichollscr https://github.com/znichollscr, we haven't published 4.3, so we could catch this before we do. Actually, I see that 4.3 is available from https://solarisheppa.geomar.de/cmip7 https://solarisheppa.geomar.de/cmip7 so we'd need to increment again to 4.4.. This dual data download sources is causing some challenges..

@berndfunke https://github.com/berndfunke as we have this ESGF publication process starting to happen quickly, would you be happy if we publish data in one place only and then link to this from the SOLARIS-HEPPA website? It seems having these data in two places is going to complicate our communication a lot

As an example, on the SOLARIS-HEPPA website you could have a link like this https://aims2.llnl.gov/search?project=input4MIPs&versionType=all&activeFacets=%7B%22mip_era%22%3A%22CMIP6Plus%22%2C%22institution_id%22%3A%22SOLARIS-HEPPA%22%2C%22source_id%22%3A%22SOLARIS-HEPPA-CMIP-4-2%22%7D which means users get the same data from a single download source?

— Reply to this email directly, view it on GitHub https://github.com/PCMDI/input4MIPs_CVs/issues/66#issuecomment-2266195437, or unsubscribe https://github.com/notifications/unsubscribe-auth/BGR2OMGETMIGVHHWPFJD5NTZPP6UTAVCNFSM6AAAAABL2EE4H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRWGE4TKNBTG4. You are receiving this because you were mentioned.

[znichollscr]

In theory yes (see screenshot), in practice, the link (e.g. https://esgf-data2.llnl.gov/thredds/dodsC/user_pub_work/input4MIPs/CMIP6Plus/CMIP/PCMDI/PCMDI-AMIP-1-1-9/ocean/fx/sftof/gn/v20230512/sftof_input4MIPs_SSTsAndSeaIce_CMIP_PCMDI-AMIP-1-1-9_gn.nc) is giving me a 400 error

[durack1]

@berndfunke yes this is totally possible, so as an example see this link. This is targeting a single file of mine (the PCMDI-AMIP-1-1-9 tosbcs) file, and two display as the project is replicated across both LLNL-ESGF and DKRZ-ESGF nodes, so two sources of data downloads

[st-bender]

@st-bender @berndfunke 1000 apologies for this, I should have caught it earlier. For your next version (whenever that is), there is one tiny other tweak to your solar data. Can you please remove the attributes datetime_end and datetime_start global attributes in all your files. They're not written correctly at the moment (the format is meant to depend on the file's frequency, a bit of a headache) and we can actually just infer them from the time axis in your data, so we can avoid sparing you the details of the frequency-dependent formatting.

We will do that for the next version. I thought that the "frequency" attribute is used to distinguish daily/monthly/...? I don't know how many of the modellers rely on specifics of the filename. If some do, we may have other issues from changing the CMIP6 accepted filename to something else.

Cheers.

[znichollscr]

I thought that the "frequency" attribute is used to distinguish daily/monthly/...?

It does, which makes the datetime_start and datetime_end attributes even more redundant (hence why we can just delete them).

I don't know how many of the modellers rely on specifics of the filename. If some do, we may have other issues from changing the CMIP6 accepted filename to something else.

Does this comment refer to #82? If yes, I will reply there to keep things in one place

[st-bender]

I don't know how many of the modellers rely on specifics of the filename. If some do, we may have other issues from changing the CMIP6 accepted filename to something else.

Does this comment refer to #82? If yes, I will reply there to keep things in one place

I was referring to the comment above, suggesting to put the files in one place and link from there. If I understand correctly, that would mean to rename the files to ESGF standards. Might just be trivial, but you never know.

[znichollscr]

I was referring to the comment above, suggesting to put the files in one place and link from there. If I understand correctly, that would mean to rename the files to ESGF standards. Might just be trivial, but you never know.

Ah ok got it. Yes definitely a tradeoff. Modelling teams would have a new filename to deal with, but it would mean that the filenames would be standardised across all the different inputs, so they wouldn't need one rule for the solar name, one rule for the volcanic name, one rule for the GHG name etc. etc.

[st-bender]

Hi @durack1

We just uploaded version 4.4 of the SOLARIS-HEPPA input files. However, the webpage still has to be updated accordingly, but the new files can already be retrieved from the same download links.

The files should now be named according to your convention, mentioned in https://github.com/PCMDI/input4MIPs_CVs/issues/82#issuecomment-2287299544, and we tried to take care that all the attributes follow the convention.

Cheers.

[znicholls]

Super thanks @st-bender. I just checked the piControl file and it looks great.

@durack1 I'll pull these onto nimbus now.

[znichollscr]

@st-bender a question to you: do you want the previous files to be retracted or are they fine, this is just new data (i.e. was there anything wrong with them or can they just stay as is)?

@durack1 raw files are in /shared/SOLARIS-HEPPA-CMIP-4-4

I've put them in the DRS in /shared/SOLARIS-HEPPA-CMIP-4-4/written-in-drs if helpful.

I think we're basically good to publish..?

[vnaik60]

Adding to Zeb's questions, it would be helpful to note the reason for this update for provenance. Thanks!

[durack1]

@st-bender @berndfunke yes, to get these published and clean things up, we'll need to know what to do with the earlier data. Was there an issue with v4.3? If yes, we'll do what we did with v4.2 retract - see here.

The other alternate is to deprecate, to keep these data published and available, but remove the "Latest" identifier, which means you have to toggle the search interface "Version Type = All" to make these older (but still valid) datasets visible - an example of this deprecated is the PCMDI-AMIP-1-1-7 data, still valid, but the temporal extensions of PCMDI-AMIP-1-1-8 and *1-1-9 versions is preferred over the earlier realease

[berndfunke]

@durack1, yes, 4.3 should be retracted. 4.4 has now the final absolute TSI scale (in consonance with NNL1) which should be used in CMIP7.

El 18 oct 2024, a las 18:01, Paul J. Durack @.***> escribió:

@st-bender https://github.com/st-bender @berndfunke https://github.com/berndfunke yes, to get these published and clean things up, we'll need to know what to do with the earlier data. Was there an issue with v4.3? If yes, we'll do what we did with v4.2 retract - see here https://aims2.llnl.gov/search?project=input4MIPs&activeFacets=%7B%22mip_era%22%3A%22CMIP6Plus%22%2C%22source_id%22%3A%22SOLARIS-HEPPA-CMIP-4-2%22%7D.

The other alternate is to deprecate, to keep these data published and available, but remove the "Latest" identifier, which means you have to toggle the search interface "Version Type = All" to make these older (but still valid) datasets visible - an example of this deprecated in the PCMDI-AMIP-1-1-7 data, still valid, but the temporal extensions of PCMDI-AMIP-1-1-8 and *1-1-9 versions is preferred over the earlier realease

— Reply to this email directly, view it on GitHub https://github.com/PCMDI/input4MIPs_CVs/issues/66#issuecomment-2422792395, or unsubscribe https://github.com/notifications/unsubscribe-auth/BGR2OMEAITL55BN6NGLYDN3Z4EWFTAVCNFSM6AAAAABL2EE4H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSG44TEMZZGU. You are receiving this because you were mentioned.

[durack1]

Ok great, retracted it is.

Can you provide a sentence or two that expands the "4.4 has now the final absolute TSI scale (in consonance with NNL1)" - what was the issue/problem with 4.3?

[berndfunke]

The release note on our page says (if already updated): Dataset version 4.4 available. update of TSI and SSI including the adoption of a new TSI scale consistent with the NNL1 model (179 ppm increase of TSI and SSI compared to version 4.3) While the preliminary TSI/SSI version in 4.3 had adapted the TSI scale from the CTIM instrument alone, the newer TSI scale is based on a TSIS-1 and CTIM composite (Coddington and Lean, 2024). This composite is also consistent with the TSI "community consensus” composite (https://spot.colorado.edu/~koppg/TSI/TSI_Composite-SIST.txt).

El 18 oct 2024, a las 18:39, Paul J. Durack @.***> escribió:

Ok great, retracted it is.

Can you provide a sentence or two that expands the "4.4 has now the final absolute TSI scale (in consonance with NNL1)" - what was the issue/problem with 4.3?

— Reply to this email directly, view it on GitHub https://github.com/PCMDI/input4MIPs_CVs/issues/66#issuecomment-2422856522, or unsubscribe https://github.com/notifications/unsubscribe-auth/BGR2OMASSCIX2YIQSNHMYZDZ4E2V3AVCNFSM6AAAAABL2EE4H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSHA2TMNJSGI. You are receiving this because you were mentioned.

[durack1]

Ok great, got these down and have been able to write out to the DRS, so that means the required metadata checks out.

@znichollscr these are in the usual place ../input4MIPs/CMIP6Plus/ ....

If I hear nothing, we'll get these into the queue for publication early next week, retracting the SOLARIS-HEPPA-CMIP-4-3 data as we go.

Pinging @sashakames so we're all in the loop

[znichollscr]

Yep files are good to go from my side too

[durack1]

And we're live - here - plus SOLARIS-HEPPA-CMIP-4-3 is now retracted.

Let's update our CVs and webpages and then close out this issue