Embed location calculations for GOES-16 in regrid_data_plane instead of -coord file

dwfncar commented 6 years ago

regrid_data_plane uses a location file to specify the location of each satellite pixel lat/lon. Instead of a static file, calculations can be done to do this on the fly for all GOES-16 scenes (FULL DISK, CONUS, MESO).

Gary Cunning created a program called GoesRnetCDF2Mdv in CVS here:
apps/ingest/src/GoesRnetCDF2Mdv

He uses the output grid (x,y) and (lat,lon) to then locate the closest satellite pixel to that location.

More information about translating between GOES-16 X,Y points and latitude/longitude can be found in the GOES-R user guide here, section 4.2.8, pp. 20-26:
https://www.goes-r.gov/products/docs/PUG-L2+-vol5.pdf [MET-1071] created by dadriaan

dwfncar commented 5 years ago

The computed lat/lon does not match exactly with the binary coordinate file:

The number of mismatch longitude: 472 out of 3,750,000:
   examples for longitude (index, value from binary, computed value, and difference:
       365, -147.609, -147.609, 0.000228882
       366, -146.585, -146.585, 0.000137329
       367, -145.882, -145.882, 0.000106812
      2863, -148.403, -148.404, 0.00050354
      2864, -146.951, -146.951, 0.000152588
      ...

The number of mismatch latitudes: 6 out of 3,750,000:
      50333, 55.9631, - 55.9633, -0.000141144
      82813, 55.4954 - 55.4957, -0.000270844
      132783, 54.7451 - 54.7453, -0.000148773
      302687, 52.3399 - 52.3401, -0.000148773
      347663, 51.7371 - 51.7374, -0.000274658
      397637, 51.0418 - 51.0419, -0.000102997

The the re-gridded output (unittest, with UW_MEAN) is little different:
< 0.02619964, 0.1537972, 0.1033263, 0.1029876, 0.1015379, 0.09170642,

0.02619964, 0.1537972, 0.1033263, 0.1029876, 0.1016333, 0.09157186,

< 0.1033003, 0.09308531, 0.09176499, 0.1244818, 0.1293496, 0.1393505,

0.1033003, 0.09312876, 0.09176499, 0.1244818, 0.1293496, 0.1393505,

< 0.1184437, 0.1228647, 0.1089977, 0.1016392, 0.07854559, 0.09684231,

0.1184437, 0.1228647, 0.1089977, 0.1015561, 0.07854559, 0.09684231,

< 0.02801347, 0.02436966, 0.157949, 0.1315053, 0.04398153, 0.06156794,

0.02801347, 0.02436966, 0.157949, 0.1322123, 0.04398153, 0.06156794,

< 0.04391601, 0.02744185, 0.02228308, 0.1701919, 0.06108617, 0.0150229,

0.04391601, 0.02744185, 0.02228308, 0.1701919, 0.06075775, 0.0150229,

< 0.223386, 0.5038977, 0.1198098, 0.1035671, 0.355451, 0.4045747,

0.223386, 0.5077441, 0.1198098, 0.1035671, 0.355451, 0.4045747,

< 0.2977682, 0.1378535, 0.07409374, 0.2349508, , , , , , , , , _,

0.2941482, 0.1378535, 0.07409374, 0.2349508, _, _, _, _, _, _, _, _, _,

by hsoh

dwfncar commented 5 years ago

Howard, John, and I met with Daniel Adriaansen 01/17 and learned the following:

The FULL DISK and CONUS scenes do not typically change. The time a change might occur would be if the satellite is moved to a new location, or if for some reason CONUS gets redefined in the future. If this ever happened, there would be a Service Change Notice or a Technical Information Bulletin from NOAA.
CONUS for GOES-East is different from CONUS for GOES-17 (which includes HI, a large chunk of the Pacific, and the U.S. West Coast).
MESO1 and MESO2 change all the time, however there are some default mesoscale windows.

Based on this, we came up with the following strategy:

The coordinates file will be included in the MET distribution for each of the GOES satellites [GOES-16 (now GOES-East) and GOES-17] for the FULL DISK and CONUS scenes. This will be 4 files with precomputed info that get included in the tarball.
Howard's logic will have regrid_data_plane check the attributes within the user's data file to see if these correspond to the known values (if FULL DISK or CONUS), and also check the extent of the domain to make sure the scene type is correct.
If MET does not find a previously stored file which matches the attributes of the data file (e.g., the scene is a new MESO1 or MESO2 window, or the coordinates do not match previously known FULL DISK or CONUS), the code will compute the grid coordinates on the fly and store this file.
The naming convention of the files files will be given some filename which might be a combination of the basic attributes (e.g., "GOES-R___center longitude of scene>"). Note that for the FULL DISK scene, the center latitude/longitude of the scene would be the same as the satellite sub-point.

Daniel will collect examples of each scene/satellite (8 files total) from today off of the SATOPS3 and put these onto /scratch/weekly so that Howard can test.

As far as the differences between the computed translated coordinates and those provided by the binary file:

Howard has analyzed the x,y coordinates of the discrepant coordinate translations and noticed that they all seem to be related to every 5th data point. So possibly, there is something going on with modulo 5, or some difference in the trigonometry implementations between Fortran and C that happens to overflow and reset every 5th point.

Until the government reopens up and we can talk to someone at NOAA, it's probably difficult or impossible to fully resolve this. One full proof way might be to get their code and compare it to ours (could be machine-dependent, compiler-dependent, or something else).

One other thing Daniel suggested as a simple qualitative validation would be to take the files he sends us, run them through MET and then use plot_data_plane and then compare to a web site like College of DuPage which keeps a 50-h running archive. This would give us a visual way to see if these differences are very important.

Overall, we decided that these differences are minor enough that we can proceed with adding this capability into the MET release, and we will keep these notes in case it becomes necessary to dig into this further. If a bug is found, could always do a bug fix later.

by jvigh

dwfncar commented 5 years ago

The -coord option is removed and the environment variable MET_GEOSTATIONARY_DATA is introduced to specify the coordinate data file.

The coordinate data file name consists with __.nc:
   - scene_id: Full, CONUS or Mesoscale
   - dx, dy: converted to integer after multiplied by 1000000
   - x_bound, y_bound: converted to integer from x_image_bounds[0] and y_image_bounds[0] after multiplied by 1000000.

grid-mapping file name consists with __to.grid_map
by hsoh

dwfncar commented 5 years ago

The coordinate NetCDF and grid mapping files are generated at MET_TMP_DIR (ot /tmp if MET_TMP_DIR is not defined)

Added two unit tests:
one for grid mapping file
the other for GOES-17 CONUS data which is 5000 by 3000. by hsoh

dtcenter / MET

Embed location calculations for GOES-16 in regrid_data_plane instead of -coord file #1071