search

JCSDA-internal / ufo-data

Tier-1 test files for ufo repository
1 stars 0 forks source link

Adding geovals file for gmi ufo ctest #365

Closed smaticka closed 1 year ago

smaticka commented 1 year ago

Description

The gmi ufo ctest requires a geovals file. One was created for the new gmi data by running the converted test data in skylab with the GOMsaver filter on.

Please merge at the same time as https://github.com/JCSDA-internal/ufo/pull/3001

Issue(s) addressed

Resolves #3035 in ufo

Dependent PR:

PR #3001 https://github.com/JCSDA-internal/ufo/pull/3001

PatNichols commented 1 year ago

@smaticka The data file is fairly large ( we need something < 500kb). Just a word to the wise that you might try to reduce the size before this goes out of draft stage. Thanks.

smaticka commented 1 year ago

@smaticka The data file is fairly large ( we need something < 500kb). Just a word to the wise that you might try to reduce the size before this goes out of draft stage. Thanks.

Oh! Thank you for catching that! I hadn't thought about that for geovals (just our other ctest data files), but makes sense that they all should be small ish.

@BenjaminRuston, @fcvdb and @fabiolrdiniz Do either of you know if I can run ncks directly on the geovals file, or if I need to reduce the original gmi testinput file, and then re-run skylab for a new geovals file?

The process followed was 1 take raw gmi file, 2 reduce it with ncks, 3 run the converter on it, 4 use the reduced raw and reduced converted files as testinput and outputs for the ctest, 5 use the converted (testouput) in skylab atm-land to generate geoval files (I had to concatenate then with ncks also).

So the question is do I need to start from step 2 and reduce it finer so that the files are compatible in the ufo ctest (my guess is yes), or can I reduce the geovals files directly?

Thanks!

smaticka commented 1 year ago

@BenjaminRuston or @fabiolrdiniz I've tried to make a smaller geovals file since @PatNichols noted that the one in this PR is too large. I tried to do so by reducing the raw data file, then converted with the new iodaconv, and re-ran in atm-land.

I'm not able to make a file smaller than 1.9MB with the process that I'm using, however, I notice in the directory jedi-bundle/ufo-data/testinput_tier_1 there are other gmi_gpm_geoval_*.nc4 files that are large, but all of the gmi_gpm_geoval_*_m.nc4 files (with an _m at the end) are smaller.

Do any of you know what process was used to make these files?

fabiolrdiniz commented 1 year ago

@smaticka, if you look into those _m files, you will notice that they have only 100 obs and were created using this command: :history = "Wed Nov 23 22:51:55 2022: ncks --cnk_dmn Location,100 -L 4 -O new.nc4 gmi_gpm_obs_2020120100_m.nc4\nTue Feb 15 00:00:23 2022: ncks -d nlocs,1,1000,10 gmi_gpm_obs_2020120100.nc4 gmi_gpm_obs_2020120100_m.nc4" ;. I hope this helps.