NCAR / ADF

A unified collection of python scripts used to generate standard plots from CAM outputs.
Creative Commons Attribution 4.0 International
34 stars 29 forks source link

Pre-made timeseries bug fix #282

Closed justin-richling closed 6 months ago

justin-richling commented 6 months ago

Currently if the user wants to use pre-made time series files, the ADF is trying to grab climo years from the given history files even if cam_ts_done variable is true because cam_hist_loc is always required. This will cause the ADF to fail as it tries to read from files that don't exist or have a different filename format that doesn't match CAM hist files.

This fix removes the requirement of cam_hist_loc so the user can now use pre-made time series files and not have to specify the cam_hist_loc.

This also introduces a check in adf_diag.py if cam_ts_done is true then it will make cam_hist_loc none if it was provided, so that the climo years are not trying to be read in from the cam_hist_loc.

This will also change the ADF behavior that requires the user to specify climo years for pre-made ts files. Now if the years are not specified, they are read in via xarrray in adf_info.py via new function get_climo_yrs_from_ts. This will print a warning message if the year range is larger than 100 because the ADF has a hard time creating climo files on the order of hundreds of years.

Additionally, this addresses the scenario if any of the test cases in a multi-case run have pre-made time series files. The user still must have the same number of entries for cam_hist_loc as the rest of variables, but they can be left blank (but not commented out) for cases that are using pre-made ts files:

Works

cam_hist_loc:
        - /glade/derecho/scratch/tilmes/archive/b.e21.B1850WmaCARMA.f19_g17.carmats16_cesm2.2.001/atm/hist/
        - 
   .
   .
   .

cam_ts_done:
        - false
        - true

Does NOT Work

cam_hist_loc:
        - /glade/derecho/scratch/tilmes/archive/b.e21.B1850WmaCARMA.f19_g17.carmats16_cesm2.2.001/atm/hist/
        #- 
   .
   .
   .

cam_ts_done:
        - false
        - true

Finally, instead of killing the ADF if the tables script encounters CMIP-like files or multiple time series files in the same directory, it just quits the tables script and continues the ADF flow.

justin-richling commented 6 months ago

@nusbaume I think I've made the suggested changes, thanks for the attention to detail!

One thing I did was expand a little more on the tables script. I've update the script so it will continue to the next variable if multiple time series files are found. I thought this would be good in an edge case where there are some variables that don't have multiple time series files, then the table can still be created.

With that, if it is a case where all variables have multiple time series files, the script will continue and still attempt to create the test case table. This will be similar to comparing against obs; the website will still produce the test case table.

I also cleaned up the case comparison section for cmip like scenario where the table is skipped, it will search for the appropriate number of tables and if there aren't enough, the case comparison table is skipped too.

I also addressed the scenario if given pre-made ts climo years are not in the found range.

If you wouldn't mind reviewing it again that would great! I tested it for all these ADF scenarios:

justin-richling commented 6 months ago

@nusbaume Thanks again, these have been fixed!