Reworking workflow to reflect changed directory structure

[charlesbluca]

Didn't have time today to fully look through the changes to the gs://cmip6 directory structure, but still opening this issue to ensure that I check this out and make the necessary updates to the workflow.

However, from a cursory glance it looks like things may not be as simple as just altering the prefix (could be subject to change as I know the big move isn't finished yet), so I think it may be useful to list out the different jobs here and go through them individually to make sure we aren't missing anything:

• CMIP__NCAR:
  • handles gs://cmip6/CMIP/NCAR/**
• CMIP__CNRM-CERFACS:
  • handles gs://cmip6/CMIP/CNRM-CERFACS/**
• CMIP__IPSL:
  • handles gs://cmip6/CMIP/IPSL/**
• CMIP__CCCma:
  • handles gs://cmip6//CMIP/CCCma/**
• CMIP__MPI-M_NCC:
  • handles gs://cmip6//CMIP/{MPI-M,NCC}/**
• CMIP__EC-Earth-Consortium_NASA-GISS:
  • handles gs://cmip6//CMIP/{EC-Earth-Consortium,NASA-GISS}/**
• CMIP__NOAA-GFDL_BCC_AWI_CSIRO_SNU_CSIRO-ARCCSS:
  • handles gs://cmip6//CMIP/{NOAA-GFDL,BCC,AWI,CSIRO,SNU,CSIRO-ARCCSS}/**
• CMIP__remainder:
  • handles all remaining directories in gs://cmip6/CMIP/**
• ScenarioMIP__CCCma:
  • handles gs://cmip6//ScenarioMIP/CCCma/**
• ScenarioMIP__CNRM-CERFACS_IPSL:
  • handles gs://cmip6//ScenarioMIP/{CNRM-CERFACS,IPSL}/**
• ScenarioMIP__EC-Earth-Consortium_MIROC_NCAR:
  • handles gs://cmip6//ScenarioMIP/{EC-Earth-Consortium,MIROC,NCAR}/**
• ScenarioMIP__MPI-M_OHC_MRI_DKRZ:
  • handles gs://cmip6//ScenarioMIP/{MPI-M,OHC,MRI,DKRZ}/**
• ScenarioMIP__remainder:
  • handles all remaining directories in gs://cmip6/ScenarioMIP/**
• DCPP__NCAR:
  • handles gs://cmip6//DCPP/NCAR/**
• DCPP__CCCma:
  • handles gs://cmip6//DCPP/CCCma/**
• DCPP__remainder:
  • handles all remaining directories in gs://cmip6/DCPP/**
• GFDL-CM26_HighResMIP_CMIP5:
  • handles gs://cmip6//{GFDL_CM2_6,HighResMIP,CMIP5}/**
• remainder:
  • handles all remaining directories in gs://cmip6/**
• catalog:
  • handles syncing of catalogs, with necessary changes made to reflect cloud provider

I'll update this tomorrow with some specifics about each job and if they can/should be changed.

cc @naomi-henderson

[naomi-henderson]

@charlesbluca , what can I do to help?

[charlesbluca]

Sorry I never followed up on this!

Essentially, just clarifying that the following changes are true:

• All the contents of CMIP have moved to CMIP6/CMIP/
• All the contents of ScenarioMIP have moved to CMIP6/ScenarioMIP/
• All remaining folders haven't been moved from their original location

Are there any specific cases where a folder under CMIP/ would've been moved to CMIP6 (e.g. CMIP/NCAR/ -> CMIP6/NCAR)?

If not, then this should just be a matter of switching/adding the prefix. If so, then we might need to go over those specific folders and make sure they are covered.

[charlesbluca]

Ah! Another big thing I forgot about:

CMIP6/ contains both CMIP/ and ScenarioMIP, but what about the subdirectories? Are those just the other folders that used to sit on the root of gs://cmip6? if so, I can probably mess around with the filters of remainder to pick up these subdirectories (as I think that once all the changes are done, the only directory it would be syncing is tracmip/).

[naomi-henderson]

Look at my pull request - I think I have made the appropriate changes

Here is the new structure of GCS:

gs://cmip6/CMIP3/
gs://cmip6/CMIP5/
gs://cmip6/CMIP6/
gs://cmip6/DCPP/
gs://cmip6/GFDL_CM2_6/
gs://cmip6/tracmip/

and here is what is now in CMIP6

gs://cmip6/CMIP6/AerChemMIP/
gs://cmip6/CMIP6/C4MIP/
gs://cmip6/CMIP6/CDRMIP/
gs://cmip6/CMIP6/CFMIP/
gs://cmip6/CMIP6/CMIP/
gs://cmip6/CMIP6/DAMIP/
gs://cmip6/CMIP6/FAFMIP/
gs://cmip6/CMIP6/GMMIP/
gs://cmip6/CMIP6/HighResMIP/
gs://cmip6/CMIP6/LS3MIP/
gs://cmip6/CMIP6/LUMIP/
gs://cmip6/CMIP6/OMIP/
gs://cmip6/CMIP6/PAMIP/
gs://cmip6/CMIP6/PMIP/
gs://cmip6/CMIP6/RFMIP/
gs://cmip6/CMIP6/ScenarioMIP/
gs://cmip6/CMIP6/old_catalogs/

[naomi-henderson]

Probably we should just ignore gs://cmip6/CMIP6/old_catalogs/ for now, it is only relevant for GCS

[charlesbluca]

Okay I just saw your branch - opened up #4 so we can do some further review and comparison.

[charlesbluca]

Thanks for the help @naomi-henderson! I gave you write access to the repo in case any other quick fixes need to be made.

Glancing through the workflow, it looks like all data should be covered now; from here, we can actively monitor the status and runtimes of the individual jobs to get a sense of if any can be merged (not really an urgent thing, but could be useful later on).

I'll also check in on #3 with any updates on changing the catalogs job to handle deleted catalogs.

[naomi-henderson]

No, thank you @charlesbluca ! I will keep my eyes on it in case any of the jobs cause trouble

[naomi-henderson]

All seems to be working with the new structure