Unable to access CLASS4 data after 2019 and only CONCEPTS is viewable

[dwayne-hart]

At this time any of the 2020 CLASS4 data files are not able to be viewed in the Ocean Navigator. Any of the CONCEPTS datasets from January 2016 to December 2019 are accessible but do not show any of the other models.

[douglatornell]

PR #797 fixes the "models other than CONCEPTS not viewable" part of this issue.

I'm looking into getting the indexing of class 4 files to be up to date to address the other part.

[douglatornell]

The the "no class 4 datasets after 2019" part is probably due to issue #757.

[jmunroe]

I am looking at what we really need when generating our list of CLASS4 files. Here's what we are currently doing:

def list_class4_files(class4_path):
    files = {f for f in Path(class4_path).glob("**/*GIOPS*profile.nc")}
    result = [
        {
            "name": datetime.datetime.strptime(class4_id.split("_")[1], "%Y%m%d").strftime("%Y-%m-%d"),
            "id": class4_id
        }
        for class4_id in sorted((f.stem for f in files), reverse=True)
    ]
    return result

Even with all the good changes from #809, it looks like we are still only considering CLASS4 files for the GIOPS model.

For reference, our CLASS4 dataset (/data/class4) is 4.9 TB over about 128175 files. I think that explain why that initial glob is so slow. But are we really just interested in files with a basename of the form _profile.nc -- there are ~13000 of those.

[jmunroe]

CLASS4 files seem to be a recurring issue with the ocean navigator. For example, here are current open issues that relate to CLASS4 files:

• class 4 delayed by nginx cache #117
• Implement API script generation for Class4 #461
• Add API endpoint to query class4 by WMO # #471
• Issue with generating class4 pickle file #757
• Class 4 functionality - not displaying all global models in comparison #764
• Unable to access CLASS4 data after 2019 and only CONCEPTS is viewable #785

It doesn't look like we have ever documented with these issues exactly what are those CLASS4 files and why do we care? The context for the problem is weather/ocean forecast verification between different ocean models and observations. From @FraserO 's presentation at a GODAE OceanView workshop in 2015 we learn

Metrics for Physical Oceanography • Class 1: Common variables on same horizontal grid • Class 2: Results interpolated to common vertical transects or virtual moorings • Class 3 Derived quantities: MeridionalOverturning SF, net heat flux • Class 4 Forecast/Analysis fields interpolated to observations

It looks like the OceanNavigator currently only supports "in situ profile T/S" but there are class4 files also available for other variables such as SST satellite, SST drifters, SSH Altimetry, and Ice Concentration (AMSR).