search

ufs-community / UFS_UTILS

Utilities for the NCEP models.
Other
21 stars 104 forks source link

Identify test opportunities, automate existing tests #217

Open edwardhartnett opened 3 years ago

edwardhartnett commented 3 years ago

@GeorgeGayno-NOAA will implement testing in the following steps 1 - Collect a set of test files, tar them up and attach to this issue. 2 - We will put on FTP site and implement script to fetch and unpack. 3 - One by one, turn on the existing tests in reg_tests.

@GeorgeGayno-NOAA will give us the first version of the test data tarball this week.

edwardhartnett commented 3 years ago

@GeorgeGayno-NOAA any progress here?

GeorgeGayno-NOAA commented 3 years ago

I created a tar ball that contains the input and baseline data for running the UFS_UTILS regression tests (on Hera): /scratch2/NCEPDEV/stmp1/George.Gayno/ufs.test.tar

It does not contain all data needed. For some tests, data from our fixed directories are used: /scratch1/NCEPDEV/global/glopara/fix/fix_sfc_climo /scratch1/NCEPDEV/global/glopara/fix/fix_orog

Some of these files are quite large.

edwardhartnett commented 3 years ago

We need all the data required to run the tests. Or we need to turn off the tests that require the largest data files. How big is the tartball with all the files?

GeorgeGayno-NOAA commented 3 years ago

We need all the data required to run the tests. Or we need to turn off the tests that require the largest data files. How big is the tartball with all the files?

The ufs.test.tar file is 52 Gb.

edwardhartnett commented 3 years ago

Then remove the biggest files and turn off those tests in automatic testing - continue to do them manually for now.

kgerheiser commented 3 years ago

Just want to chime in that I think having some sort of more manageable test case (a few GBs) would be nice so it can be run on personal machines. Maybe an option to not use the C192 tests? Would that have a significant effect on the original 52 GB file size?

GeorgeGayno-NOAA commented 3 years ago

Just want to chime in that I think having some sort of more manageable test case (a few GBs) would be nice so it can be run on personal machines. Maybe an option to not use the C192 tests? Would that have a significant effect on the original 52 GB file size?

The largest files are associated with the input data. To get an idea of the file sizes see this directory on Venus: /gpfs/dell2/emc/modeling/noscrub/George.Gayno/ufs_utils.git/reg_tests/chgres_cube/input_data

edwardhartnett commented 3 years ago

Can we start with just one test file that is less than 100 MB?

GeorgeGayno-NOAA commented 3 years ago

@GeorgeGayno-NOAA will implement testing in the following steps 1 - Collect a set of test files, tar them up and attach to this issue. 2 - We will put on FTP site and implement script to fetch and unpack. 3 - One by one, turn on the existing tests in reg_tests.

@GeorgeGayno-NOAA will give us the first version of the test data tarball this week.

Which FTP site will we use for our repos?

edwardhartnett commented 3 years ago

I've sent an email with the ftp site info...

GeorgeGayno-NOAA commented 3 years ago

@KateFriedman-NOAA will manage the ftp site: https://ftp.emc.ncep.noaa.gov/static_files/public/

edwardhartnett commented 3 years ago

What is the progress here? Do we have a test running yet?

GeorgeGayno-NOAA commented 3 years ago

What is the progress here? Do we have a test running yet?

I have not had a chance to work on this issue.

edwardhartnett commented 3 years ago

OK, this task needs to move up the priority list. Kyle, Lin, and I will now be assisting writing tests for UFS_UTILS.

@GeorgeGayno-NOAA did you send any sample data to Kate for the FTP site?

kgerheiser commented 3 years ago

I downloaded all the UFS_UTILS test data. These are the utils and the size of the test data (uncompressed).

snow2mdl: 31.9 MB

ice_blend: 22.2 MB

grid_gen: 1.4 GB

global_cycle: 3.67 GB

chgres_cube: 50 GB

chgres_cube.public.v2: 50 GB

edwardhartnett commented 3 years ago

OK, can you send the snow2mdl and ice_blend data to Kate to put on the FTP site? That will be our first round, since it's low-hanging fruit.

How can we test chres_cube without that 50 GB file. (Is it one file? Or multiple? What is there, exactly, can you post an ls -l?)

kgerheiser commented 3 years ago

For chgres_cube, there are sub tests for different kinds of I/O. It's not all one big file (e.g. nemsio, netcdf).

Test directory layout

``` └── [ 98G] reg_tests ├── [ 46G] chgres_cube │   ├── [3.6G] baseline_data │   │   ├── [709M] c192_fv3_history │   │   │   ├── [1.8K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [100M] out.atm.tile1.nc │   │   │   ├── [100M] out.atm.tile2.nc │   │   │   ├── [100M] out.atm.tile3.nc │   │   │   ├── [100M] out.atm.tile4.nc │   │   │   ├── [100M] out.atm.tile5.nc │   │   │   ├── [100M] out.atm.tile6.nc │   │   │   ├── [ 18M] out.sfc.tile1.nc │   │   │   ├── [ 18M] out.sfc.tile2.nc │   │   │   ├── [ 18M] out.sfc.tile3.nc │   │   │   ├── [ 18M] out.sfc.tile4.nc │   │   │   ├── [ 18M] out.sfc.tile5.nc │   │   │   └── [ 18M] out.sfc.tile6.nc │   │   ├── [907M] c192_gfs_grib2 │   │   │   ├── [1.5K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [138M] out.atm.tile1.nc │   │   │   ├── [138M] out.atm.tile2.nc │   │   │   ├── [138M] out.atm.tile3.nc │   │   │   ├── [138M] out.atm.tile4.nc │   │   │   ├── [138M] out.atm.tile5.nc │   │   │   ├── [138M] out.atm.tile6.nc │   │   │   ├── [ 13M] out.sfc.tile1.nc │   │   │   ├── [ 13M] out.sfc.tile2.nc │   │   │   ├── [ 13M] out.sfc.tile3.nc │   │   │   ├── [ 13M] out.sfc.tile4.nc │   │   │   ├── [ 13M] out.sfc.tile5.nc │   │   │   └── [ 13M] out.sfc.tile6.nc │   │   ├── [898M] c192_gfs_grib2.6bf5373 │   │   │   ├── [1.5K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [138M] out.atm.tile1.nc │   │   │   ├── [138M] out.atm.tile2.nc │   │   │   ├── [138M] out.atm.tile3.nc │   │   │   ├── [138M] out.atm.tile4.nc │   │   │   ├── [138M] out.atm.tile5.nc │   │   │   ├── [138M] out.atm.tile6.nc │   │   │   ├── [ 12M] out.sfc.tile1.nc │   │   │   ├── [ 12M] out.sfc.tile2.nc │   │   │   ├── [ 12M] out.sfc.tile3.nc │   │   │   ├── [ 12M] out.sfc.tile4.nc │   │   │   ├── [ 12M] out.sfc.tile5.nc │   │   │   └── [ 12M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_nemsio │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_netcdf │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_restart │   │   │   ├── [2.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [178M] c96_gfs_nemsio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [170M] c96_gfs_sigio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [3.2M] out.sfc.tile1.nc │   │   │   ├── [3.2M] out.sfc.tile2.nc │   │   │   ├── [3.2M] out.sfc.tile3.nc │   │   │   ├── [3.2M] out.sfc.tile4.nc │   │   │   ├── [3.2M] out.sfc.tile5.nc │   │   │   └── [3.2M] out.sfc.tile6.nc │   │   └── [180M] c96_regional │   │   ├── [1.3K] fort.41 │   │   ├── [ 12M] gfs.bndy.nc │   │   ├── [7.4K] gfs_ctrl.nc │   │   ├── [149M] out.atm.tile1.nc │   │   └── [ 19M] out.sfc.tile1.nc │   ├── [388M] fix │   │   ├── [ 65M] C192 │   │   │   ├── [7.9M] C192_grid.tile1.nc │   │   │   ├── [7.9M] C192_grid.tile2.nc │   │   │   ├── [7.9M] C192_grid.tile3.nc │   │   │   ├── [7.9M] C192_grid.tile4.nc │   │   │   ├── [7.9M] C192_grid.tile5.nc │   │   │   ├── [7.9M] C192_grid.tile6.nc │   │   │   ├── [ 22K] C192_mosaic.nc │   │   │   ├── [2.8M] C192_oro_data.tile1.nc │   │   │   ├── [2.8M] C192_oro_data.tile2.nc │   │   │   ├── [2.8M] C192_oro_data.tile3.nc │   │   │   ├── [2.8M] C192_oro_data.tile4.nc │   │   │   ├── [2.8M] C192_oro_data.tile5.nc │   │   │   ├── [2.8M] C192_oro_data.tile6.nc │   │   │   └── [1.6K] fix_sfc │   │   ├── [257M] C384 │   │   │   ├── [ 32M] C384_grid.tile1.nc │   │   │   ├── [ 32M] C384_grid.tile2.nc │   │   │   ├── [ 32M] C384_grid.tile3.nc │   │   │   ├── [ 32M] C384_grid.tile4.nc │   │   │   ├── [ 32M] C384_grid.tile5.nc │   │   │   ├── [ 32M] C384_grid.tile6.nc │   │   │   ├── [ 22K] C384_mosaic.nc │   │   │   ├── [ 11M] C384_oro_data.tile1.nc │   │   │   ├── [ 11M] C384_oro_data.tile2.nc │   │   │   ├── [ 11M] C384_oro_data.tile3.nc │   │   │   ├── [ 11M] C384_oro_data.tile4.nc │   │   │   ├── [ 11M] C384_oro_data.tile5.nc │   │   │   └── [ 11M] C384_oro_data.tile6.nc │   │   ├── [ 16M] C96 │   │   │   ├── [2.0M] C96_grid.tile1.nc │   │   │   ├── [2.0M] C96_grid.tile2.nc │   │   │   ├── [2.0M] C96_grid.tile3.nc │   │   │   ├── [2.0M] C96_grid.tile4.nc │   │   │   ├── [2.0M] C96_grid.tile5.nc │   │   │   ├── [2.0M] C96_grid.tile6.nc │   │   │   ├── [ 22K] C96_mosaic.nc │   │   │   ├── [736K] C96_oro_data.tile1.nc │   │   │   ├── [736K] C96_oro_data.tile2.nc │   │   │   ├── [736K] C96_oro_data.tile3.nc │   │   │   ├── [736K] C96_oro_data.tile4.nc │   │   │   ├── [736K] C96_oro_data.tile5.nc │   │   │   ├── [736K] C96_oro_data.tile6.nc │   │   │   └── [1.6K] fix_sfc │   │   └── [ 50M] C96.regional │   │   ├── [8.7M] C96_grid.tile7.halo0.nc │   │   ├── [9.2M] C96_grid.tile7.halo3.nc │   │   ├── [9.4M] C96_grid.tile7.halo4.nc │   │   ├── [9.4M] C96_grid.tile7.nc │   │   ├── [8.4K] C96_mosaic.nc │   │   ├── [3.1M] C96_oro_data.tile7.halo0.nc │   │   ├── [3.3M] C96_oro_data.tile7.halo3.nc │   │   ├── [3.3M] C96_oro_data.tile7.halo4.nc │   │   ├── [3.3M] C96_oro_data.tile7.nc │   │   └── [ 832] fix_sfc │   └── [ 42G] input_data │   ├── [153M] fv3.history │   │   ├── [ 20M] dynf000.tile1.nc │   │   ├── [ 20M] dynf000.tile2.nc │   │   ├── [ 20M] dynf000.tile3.nc │   │   ├── [ 20M] dynf000.tile4.nc │   │   ├── [ 20M] dynf000.tile5.nc │   │   ├── [ 20M] dynf000.tile6.nc │   │   ├── [5.3M] phyf000.tile1.nc │   │   ├── [5.3M] phyf000.tile2.nc │   │   ├── [5.3M] phyf000.tile3.nc │   │   ├── [5.3M] phyf000.tile4.nc │   │   ├── [5.3M] phyf000.tile5.nc │   │   └── [5.3M] phyf000.tile6.nc │   ├── [ 18G] fv3.nemsio │   │   ├── [ 16G] gfs.t12z.atmf000.nemsio │   │   └── [2.6G] gfs.t12z.sfcf000.nemsio │   ├── [7.4G] fv3.netcdf │   │   ├── [6.5G] gfs.t00z.atmf000.nc │   │   └── [999M] gfs.t00z.sfcf000.nc │   ├── [3.8G] fv3.restart │   │   ├── [ 17K] 20190706.120000.fv_core.res.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile1.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile2.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile3.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile4.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile5.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile6.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile1.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile2.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile3.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile4.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile5.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile6.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile1.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile2.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile3.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile4.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile5.nc │   │   └── [ 70M] 20190706.120000.sfc_data.tile6.nc │   ├── [ 88M] gfs.grib2 │   │   └── [ 88M] gfs.t00z.pgrb2.0p50.f000 │   ├── [8.0G] gfs.nemsio │   │   ├── [6.9G] gfnanl.gdas.2017071700 │   │   ├── [378M] nsnanl.gdas.2017071700 │   │   ├── [ 14K] sfc.ctl │   │   └── [828M] sfnanl.gdas.2017071700 │   └── [4.2G] gfs.sigio │   ├── [3.4G] gdas.t00z.sanl │   └── [792M] gdas.t00z.sfcanl ├── [ 48G] chgres_cube.public.v2 │   ├── [2.8G] baseline_data │   │   ├── [709M] c192_fv3_history │   │   │   ├── [1.8K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [100M] out.atm.tile1.nc │   │   │   ├── [100M] out.atm.tile2.nc │   │   │   ├── [100M] out.atm.tile3.nc │   │   │   ├── [100M] out.atm.tile4.nc │   │   │   ├── [100M] out.atm.tile5.nc │   │   │   ├── [100M] out.atm.tile6.nc │   │   │   ├── [ 18M] out.sfc.tile1.nc │   │   │   ├── [ 18M] out.sfc.tile2.nc │   │   │   ├── [ 18M] out.sfc.tile3.nc │   │   │   ├── [ 18M] out.sfc.tile4.nc │   │   │   ├── [ 18M] out.sfc.tile5.nc │   │   │   └── [ 18M] out.sfc.tile6.nc │   │   ├── [907M] c192_gfs_grib2 │   │   │   ├── [1.5K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [138M] out.atm.tile1.nc │   │   │   ├── [138M] out.atm.tile2.nc │   │   │   ├── [138M] out.atm.tile3.nc │   │   │   ├── [138M] out.atm.tile4.nc │   │   │   ├── [138M] out.atm.tile5.nc │   │   │   ├── [138M] out.atm.tile6.nc │   │   │   ├── [ 13M] out.sfc.tile1.nc │   │   │   ├── [ 13M] out.sfc.tile2.nc │   │   │   ├── [ 13M] out.sfc.tile3.nc │   │   │   ├── [ 13M] out.sfc.tile4.nc │   │   │   ├── [ 13M] out.sfc.tile5.nc │   │   │   └── [ 13M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_nemsio │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_netcdf │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_restart │   │   │   ├── [2.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [178M] c96_gfs_nemsio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [170M] c96_gfs_sigio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [3.2M] out.sfc.tile1.nc │   │   │   ├── [3.2M] out.sfc.tile2.nc │   │   │   ├── [3.2M] out.sfc.tile3.nc │   │   │   ├── [3.2M] out.sfc.tile4.nc │   │   │   ├── [3.2M] out.sfc.tile5.nc │   │   │   └── [3.2M] out.sfc.tile6.nc │   │   └── [180M] c96_regional │   │   ├── [1.3K] fort.41 │   │   ├── [ 12M] gfs.bndy.nc │   │   ├── [7.4K] gfs_ctrl.nc │   │   ├── [149M] out.atm.tile1.nc │   │   └── [ 19M] out.sfc.tile1.nc │   ├── [2.8G] baseline_data.0b68195 │   │   ├── [709M] c192_fv3_history │   │   │   ├── [1.8K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [100M] out.atm.tile1.nc │   │   │   ├── [100M] out.atm.tile2.nc │   │   │   ├── [100M] out.atm.tile3.nc │   │   │   ├── [100M] out.atm.tile4.nc │   │   │   ├── [100M] out.atm.tile5.nc │   │   │   ├── [100M] out.atm.tile6.nc │   │   │   ├── [ 18M] out.sfc.tile1.nc │   │   │   ├── [ 18M] out.sfc.tile2.nc │   │   │   ├── [ 18M] out.sfc.tile3.nc │   │   │   ├── [ 18M] out.sfc.tile4.nc │   │   │   ├── [ 18M] out.sfc.tile5.nc │   │   │   └── [ 18M] out.sfc.tile6.nc │   │   ├── [907M] c192_gfs_grib2 │   │   │   ├── [1.5K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [138M] out.atm.tile1.nc │   │   │   ├── [138M] out.atm.tile2.nc │   │   │   ├── [138M] out.atm.tile3.nc │   │   │   ├── [138M] out.atm.tile4.nc │   │   │   ├── [138M] out.atm.tile5.nc │   │   │   ├── [138M] out.atm.tile6.nc │   │   │   ├── [ 13M] out.sfc.tile1.nc │   │   │   ├── [ 13M] out.sfc.tile2.nc │   │   │   ├── [ 13M] out.sfc.tile3.nc │   │   │   ├── [ 13M] out.sfc.tile4.nc │   │   │   ├── [ 13M] out.sfc.tile5.nc │   │   │   └── [ 13M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_nemsio │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_netcdf │   │   │   ├── [1.4K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [232M] c96_fv3_restart │   │   │   ├── [2.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 34M] out.atm.tile1.nc │   │   │   ├── [ 34M] out.atm.tile2.nc │   │   │   ├── [ 34M] out.atm.tile3.nc │   │   │   ├── [ 34M] out.atm.tile4.nc │   │   │   ├── [ 34M] out.atm.tile5.nc │   │   │   ├── [ 34M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [178M] c96_gfs_nemsio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [4.5M] out.sfc.tile1.nc │   │   │   ├── [4.5M] out.sfc.tile2.nc │   │   │   ├── [4.5M] out.sfc.tile3.nc │   │   │   ├── [4.5M] out.sfc.tile4.nc │   │   │   ├── [4.5M] out.sfc.tile5.nc │   │   │   └── [4.5M] out.sfc.tile6.nc │   │   ├── [170M] c96_gfs_sigio │   │   │   ├── [1.3K] fort.41 │   │   │   ├── [7.4K] gfs_ctrl.nc │   │   │   ├── [ 25M] out.atm.tile1.nc │   │   │   ├── [ 25M] out.atm.tile2.nc │   │   │   ├── [ 25M] out.atm.tile3.nc │   │   │   ├── [ 25M] out.atm.tile4.nc │   │   │   ├── [ 25M] out.atm.tile5.nc │   │   │   ├── [ 25M] out.atm.tile6.nc │   │   │   ├── [3.2M] out.sfc.tile1.nc │   │   │   ├── [3.2M] out.sfc.tile2.nc │   │   │   ├── [3.2M] out.sfc.tile3.nc │   │   │   ├── [3.2M] out.sfc.tile4.nc │   │   │   ├── [3.2M] out.sfc.tile5.nc │   │   │   └── [3.2M] out.sfc.tile6.nc │   │   └── [180M] c96_regional │   │   ├── [1.3K] fort.41 │   │   ├── [ 12M] gfs.bndy.nc │   │   ├── [7.4K] gfs_ctrl.nc │   │   ├── [149M] out.atm.tile1.nc │   │   └── [ 19M] out.sfc.tile1.nc │   ├── [388M] fix │   │   ├── [ 65M] C192 │   │   │   ├── [7.9M] C192_grid.tile1.nc │   │   │   ├── [7.9M] C192_grid.tile2.nc │   │   │   ├── [7.9M] C192_grid.tile3.nc │   │   │   ├── [7.9M] C192_grid.tile4.nc │   │   │   ├── [7.9M] C192_grid.tile5.nc │   │   │   ├── [7.9M] C192_grid.tile6.nc │   │   │   ├── [ 22K] C192_mosaic.nc │   │   │   ├── [2.8M] C192_oro_data.tile1.nc │   │   │   ├── [2.8M] C192_oro_data.tile2.nc │   │   │   ├── [2.8M] C192_oro_data.tile3.nc │   │   │   ├── [2.8M] C192_oro_data.tile4.nc │   │   │   ├── [2.8M] C192_oro_data.tile5.nc │   │   │   ├── [2.8M] C192_oro_data.tile6.nc │   │   │   └── [1.6K] fix_sfc │   │   ├── [257M] C384 │   │   │   ├── [ 32M] C384_grid.tile1.nc │   │   │   ├── [ 32M] C384_grid.tile2.nc │   │   │   ├── [ 32M] C384_grid.tile3.nc │   │   │   ├── [ 32M] C384_grid.tile4.nc │   │   │   ├── [ 32M] C384_grid.tile5.nc │   │   │   ├── [ 32M] C384_grid.tile6.nc │   │   │   ├── [ 22K] C384_mosaic.nc │   │   │   ├── [ 11M] C384_oro_data.tile1.nc │   │   │   ├── [ 11M] C384_oro_data.tile2.nc │   │   │   ├── [ 11M] C384_oro_data.tile3.nc │   │   │   ├── [ 11M] C384_oro_data.tile4.nc │   │   │   ├── [ 11M] C384_oro_data.tile5.nc │   │   │   └── [ 11M] C384_oro_data.tile6.nc │   │   ├── [ 16M] C96 │   │   │   ├── [2.0M] C96_grid.tile1.nc │   │   │   ├── [2.0M] C96_grid.tile2.nc │   │   │   ├── [2.0M] C96_grid.tile3.nc │   │   │   ├── [2.0M] C96_grid.tile4.nc │   │   │   ├── [2.0M] C96_grid.tile5.nc │   │   │   ├── [2.0M] C96_grid.tile6.nc │   │   │   ├── [ 22K] C96_mosaic.nc │   │   │   ├── [736K] C96_oro_data.tile1.nc │   │   │   ├── [736K] C96_oro_data.tile2.nc │   │   │   ├── [736K] C96_oro_data.tile3.nc │   │   │   ├── [736K] C96_oro_data.tile4.nc │   │   │   ├── [736K] C96_oro_data.tile5.nc │   │   │   ├── [736K] C96_oro_data.tile6.nc │   │   │   └── [1.6K] fix_sfc │   │   └── [ 50M] C96.regional │   │   ├── [8.7M] C96_grid.tile7.halo0.nc │   │   ├── [9.2M] C96_grid.tile7.halo3.nc │   │   ├── [9.4M] C96_grid.tile7.halo4.nc │   │   ├── [9.4M] C96_grid.tile7.nc │   │   ├── [8.4K] C96_mosaic.nc │   │   ├── [3.1M] C96_oro_data.tile7.halo0.nc │   │   ├── [3.3M] C96_oro_data.tile7.halo3.nc │   │   ├── [3.3M] C96_oro_data.tile7.halo4.nc │   │   ├── [3.3M] C96_oro_data.tile7.nc │   │   └── [ 832] fix_sfc │   └── [ 42G] input_data │   ├── [153M] fv3.history │   │   ├── [ 20M] dynf000.tile1.nc │   │   ├── [ 20M] dynf000.tile2.nc │   │   ├── [ 20M] dynf000.tile3.nc │   │   ├── [ 20M] dynf000.tile4.nc │   │   ├── [ 20M] dynf000.tile5.nc │   │   ├── [ 20M] dynf000.tile6.nc │   │   ├── [5.3M] phyf000.tile1.nc │   │   ├── [5.3M] phyf000.tile2.nc │   │   ├── [5.3M] phyf000.tile3.nc │   │   ├── [5.3M] phyf000.tile4.nc │   │   ├── [5.3M] phyf000.tile5.nc │   │   └── [5.3M] phyf000.tile6.nc │   ├── [ 18G] fv3.nemsio │   │   ├── [ 16G] gfs.t12z.atmf000.nemsio │   │   └── [2.6G] gfs.t12z.sfcf000.nemsio │   ├── [7.4G] fv3.netcdf │   │   ├── [6.5G] gfs.t00z.atmf000.nc │   │   └── [999M] gfs.t00z.sfcf000.nc │   ├── [3.8G] fv3.restart │   │   ├── [ 17K] 20190706.120000.fv_core.res.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile1.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile2.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile3.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile4.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile5.nc │   │   ├── [289M] 20190706.120000.fv_core.res.tile6.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile1.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile2.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile3.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile4.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile5.nc │   │   ├── [288M] 20190706.120000.fv_tracer.res.tile6.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile1.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile2.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile3.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile4.nc │   │   ├── [ 70M] 20190706.120000.sfc_data.tile5.nc │   │   └── [ 70M] 20190706.120000.sfc_data.tile6.nc │   ├── [ 88M] gfs.grib2 │   │   └── [ 88M] gfs.t00z.pgrb2.0p50.f000 │   ├── [8.0G] gfs.nemsio │   │   ├── [6.9G] gfnanl.gdas.2017071700 │   │   ├── [378M] nsnanl.gdas.2017071700 │   │   ├── [ 14K] sfc.ctl │   │   └── [828M] sfnanl.gdas.2017071700 │   └── [4.2G] gfs.sigio │   ├── [3.4G] gdas.t00z.sanl │   └── [792M] gdas.t00z.sfcanl ├── [3.3G] global_cycle │   ├── [1.6G] baseline_data │   │   ├── [279M] 20190730.000000.sfcanl_data.tile1.nc │   │   ├── [279M] 20190730.000000.sfcanl_data.tile2.nc │   │   ├── [279M] 20190730.000000.sfcanl_data.tile3.nc │   │   ├── [279M] 20190730.000000.sfcanl_data.tile4.nc │   │   ├── [279M] 20190730.000000.sfcanl_data.tile5.nc │   │   └── [279M] 20190730.000000.sfcanl_data.tile6.nc │   └── [1.7G] input_data │   ├── [279M] 20190730.000000.sfc_data.tile1.nc │   ├── [279M] 20190730.000000.sfc_data.tile2.nc │   ├── [279M] 20190730.000000.sfc_data.tile3.nc │   ├── [279M] 20190730.000000.sfc_data.tile4.nc │   ├── [279M] 20190730.000000.sfc_data.tile5.nc │   ├── [279M] 20190730.000000.sfc_data.tile6.nc │   ├── [ 10M] gdas.t00z.dtfanl.nc │   ├── [ 14M] gdas.t00z.rtgssthr.grb │   ├── [8.9M] gdas.t00z.seaice.5min.blend.grb │   └── [5.3M] gdas.t00z.snogrb_t1534.3072.1536 ├── [741M] grid_gen │   └── [741M] baseline_data │   ├── [ 16M] c96.uniform │   │   ├── [2.0M] C96_grid.tile1.nc │   │   ├── [2.0M] C96_grid.tile2.nc │   │   ├── [2.0M] C96_grid.tile3.nc │   │   ├── [2.0M] C96_grid.tile4.nc │   │   ├── [2.0M] C96_grid.tile5.nc │   │   ├── [2.0M] C96_grid.tile6.nc │   │   ├── [ 23K] C96_mosaic.nc │   │   ├── [740K] C96_oro_data.tile1.nc │   │   ├── [740K] C96_oro_data.tile2.nc │   │   ├── [740K] C96_oro_data.tile3.nc │   │   ├── [740K] C96_oro_data.tile4.nc │   │   ├── [740K] C96_oro_data.tile5.nc │   │   ├── [740K] C96_oro_data.tile6.nc │   │   └── [1.6K] fix_sfc │   ├── [687M] esg.regional │   │   ├── [167M] C3113_grid.tile7.halo0.nc │   │   ├── [169M] C3113_grid.tile7.halo3.nc │   │   ├── [170M] C3113_grid.tile7.halo4.nc │   │   ├── [ 18K] C3113_mosaic.nc │   │   ├── [ 60M] C3113_oro_data.tile7.halo0.nc │   │   ├── [ 60M] C3113_oro_data.tile7.halo3.nc │   │   ├── [ 61M] C3113_oro_data.tile7.halo4.nc │   │   └── [ 576] fix_sfc │   └── [ 37M] gfdl.regional │   ├── [8.7M] C424_grid.tile7.halo0.nc │   ├── [9.2M] C424_grid.tile7.halo3.nc │   ├── [9.4M] C424_grid.tile7.halo4.nc │   ├── [9.2K] C424_mosaic.nc │   ├── [3.1M] C424_oro_data.tile7.halo0.nc │   ├── [3.3M] C424_oro_data.tile7.halo3.nc │   ├── [3.3M] C424_oro_data.tile7.halo4.nc │   └── [ 576] fix_sfc ├── [ 19M] ice_blend │   ├── [8.9M] baseline_data │   │   └── [8.9M] seaice.5min.blend │   └── [ 10M] input_data │   ├── [9.8M] imssnow96.grib2.gdas.2018120618 │   └── [503K] seaice.5min.grib2.gdas.2018120618 └── [ 29M] snow2mdl ├── [5.3M] baseline_data │   └── [5.3M] snogrb_model └── [ 23M] input_data ├── [2.2M] NPR.SNWN.SP.S1200.MESH16 ├── [2.2M] NPR.SNWS.SP.S1200.MESH16 └── [ 19M] imssnow96.grb ```

kgerheiser commented 3 years ago

I converted the snow2mdl test to use CMake and it successfully ran on my desktop computer.

https://github.com/kgerheiser/UFS_UTILS/commit/0d5bee91709e359b11df26047e4a2aa0d54affb7

I kept everything the same, but I added some CMake in reg_tests and made a driver.cmake.sh script that uses local files. It presumes a data folder and the fix files in fix.

For snow2mdl these files total 38 MB. I think I can do this for a bunch of the tests.

Update:

ice_blend also works