added historical metadata for 21 v2_LE ensembles

[TonyB9000]

Still want to leave this branch open for the v2 LE ssp370 metadata - one I figure how to create it

[TonyB9000]

@chengzhuzhang I just added the 21 ssp370 metadata files. Please check for issues. References will likely change, of course.

[chengzhuzhang]

On thing i notice, the parent_variant_label, in ssp370_rxi1p1f1.json, should match rxi1p1f1 for each historical runs that it branch from.

[TonyB9000]

@chengzhuzhang Good catch - I missed that. (Also, in the table above, the v1_LE_hist bpip should be 3560, not 35600.)

My methodology was to difference:

(A) v1_LE_historical versus v2_LE_historical  (BIG differences)
(B) v1_LE_historical versus v1_LE_ssp370 (minor differences)

Given that information, it made sense to assume that the eventual diff:

v2_LE_historical versus v2_LE_ssp370

would exhibit similar minor differences. Thus I began by copying the v2_LE_historical metadata files to v2_LE_ssp370 files, and then using the (B) differences to inform me what changes need to be made. (See the email "Git Help with Metadata Repo", Tues Aug 29 1:23 PM.)

[TonyB9000]

Also, there are no "v2_LE_historical" ensemble 1-5 sets. Those were the original v2_historical. Which perhaps brings up the an issue with the mysterious Epics&Tasks (Phase 3 Data Publication: Publish additional v1 LE historical + ssp370 (5 ensembles)). To my knowledge, I was never given a description of this tasking.

[chengzhuzhang]

@TonyB9000 those additional ensembles will go after we finish current tasks at hand.

[TonyB9000]

@chengzhuzhang I note that none of the metadata files (v1 or v2, historical or ssp370) have their own "variant label" specified (except in the file-name), only "parent_variant_label". Are you saying that the ssp370 sets should have "parent" matching the variant label of the matching historical ensemble? In that case, all of the v1 sets are wrong as well ... :( We can fix the v1 data files (1) using Charlie's magic, but all would need to be republished...

[chengzhuzhang]

yes, "parent_variant_label" v1 sets need to be fixed.. :(

The variant label is defined, like following:

  "realization_index": "10",

  "initialization_index": "2",

  "physics_index": "2",

  "forcing_index": "1",

[TonyB9000]

@chengzhuzhang Understood. It was the relationship between ssp370 sets and the historical sets that I was unaware of (I didn't know that the ssp370 ensemble N was derivative of historical ensemble N.)

(Another major repair job. At least we caught this before generating the v2 LE ssp370 CMIP6 sets)

[TonyB9000]

Assessment: As far as the individual metadata files go:

v1_LE (both historical and ssp370) have correct realization, initialization, physics and forcing indexes (N,2,2,1). Only the "parent_variant_label" needs to be fixed (still a major republication effort.)

v2_LE (both historical and ssp370) have correct realization, initialization, physics and forcing indexes (N,1,1,1), Again, only the "parent_variant_label" needs to be fixed.

I'll fix, and push both now, and add the retraction/republication of v1_LE_ssp370 data to the task queue...

[chengzhuzhang]

v2_LE (both historical and ssp370) have correct realization, initialization, physics and forcing indexes (N,1,1,1), Again, only the "parent_variant_label" needs to be fixed.

the parent_variant_label for v1/2_LE historical is correct. only the v1/2_ssp are branched from each of their parent historical runs.

[TonyB9000]

@chengzhuzhang Right. I should have said "only the ssp370 parent_variant_labels" need correction (v1 and v2)

(Sorry - was listing the wrong directory - all have similar names)

[TonyB9000]

All v2 ssp370 are fixed on this branch. (v1_LE ssp370 are fixed on another branch).