Make GOES WMS Great Again

akrherz commented 5 years ago

I shutdown GOES East WMS services with the demise of GINI products when GOES-R went live. Today, Unidata announced some fancy netcdf4 products which would make my processing life much easier. In fact, perhaps we can make GOES WMS Great Again.

``` IMPORTANT notice of major changes to the IDD NIMAGE datastream; please read. First, we apologies for cross posting this announcement to multiple lists, but we want to make sure that all users that will/could be affected are notified. Second, we realize that the following message is long, but please read it all since it contains important information that will affect sites that are or will be receiving imagery products in the IDD NIMAGE feed. What: We will be transitioning the IDD NIMAGE datastream to one that contains all of the GOES-16 and GOES-17 Level 2 image products that are currently being being sent in the IDD NOTHER and HDS datastreams. When: It is our desire to begin distributing a reconstituted NIMAGE feed from our top level IDD relay clusters (idd.unidata.ucar.edu and iddb.unidata.ucar.edu) in the mid-June time frame. We will postpone turning on this feed if/when end-users indicate that they need more time to prepare for the change. Impact: The volume of data available in the NIMAGE feed will increase from an average of about 100 MB/hr to an average that is close to 7 GB/hr, a 70-fold increase! Reason for the change: NOAA has indicated that operation of GOES-15 will likely cease in early-mid July: https://www.goes-r.gov/users/transitionToOperations17.html Since GOES-East imagery in GINI format was removed from the NOAAPort SBN after GOES-13 was replaced as GOES-East by GOES-16 and then decommissioned, we believe that it is most likely that all GINI imagery will cease to be distributed in NOAAPort in the mid-July time frame. Since the NIMAGE IDD data feed has been populated solely by the GINI imagery distributed in NOAAPort, this feed will cease to contain any data. Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline Imager) Level 2 product tiles (pieces of full image scenes), and those tiles are being distributed in the IDD NOTHER data feed. To date, these image tiles would need to be stitched together by end-users before they were fully usable in display and analysis applications available from Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy (this is not strictly true for AWIPS)). In addition to the ABI Level 2 product tiles, 19 other Level 2 products are also being distributed in NOAAPort, and we have been making those products available in the IDD NOTHER and HDS data feeds since their addition to NOAAPort. Like the ABI Level 2 product tiles, the other Level 2 products need some processing before they are usable in end-user applications - they need to have the NOAAPort header and footer stripped off to leave the underlying netCDF4 files. We believe that it will be much easier for end-users, who get their NOAAPort-delivered Level 2 images and products via the IDD, if we reconstitute full images scenes from ABI tiles and strip NOAAPort broadcast headers and footers here in Unidata and then distributed the resulting netCDF4 images/products in a reconstituted NIMAGE IDD data feed. Also, reconstituting the NIMAGE feed will allow us to make other, value-added GOES-16/17 products available to the community. The first of these products is GLM imagery/grids being produced at Texas Tech by community member Eric Bruning. Information on the imagery and products that are currently contained in and those that will be added to the reconstituted NIMAGE feed can be found in: https://www.unidata.ucar.edu/data/nimage.html We have been using all of the products in this new incarnation of the NIMAGE feed internally for the past few months to test their viability in the various application packages that we provide and support. We are satisfied that transition to use of these products by end users will be straightforward, but it will require that sites install and transition to use of the latest of each package that we make available. Important things to note: - the reconstitution of the NIMAGE feed comes at a cost The volume of data available will increase from an average of about 100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase! Sites that do not want to receive any/all of the processed GOES-16/17 imagery/products in the new NIMAGE feed need to make sure that their LDM configuration file REQUEST(s) do not include products that they do not want. The current GOES-15 GINI imagery that is being distributed in NOAAPort all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting all of the current NIMAGE feed via a REQUEST that looks like: REQUEST NIMAGE ".*" idd.unidata.ucar.edu and that either do not want or can not handle the increased data volume should immediately change their REQUEST to something like: REQUEST NIMAGE "satz" idd.unidata.ucar.edu Sites that do want to receive all of the imagery that will be available in the reconstituted NIMAGE feed and whose network connection can handle the increased volume can leave their REQUEST(s) as is, but this decision should not be taken lightly given the 70-fold increase in data volume! Sites wanting to receive some of the data from the reconstituted NIMAGE feed are advised to read through the NIMAGE web page listed above to become familiar with how the LDM/IDD Product IDs will look and then re-fashion their feed REQUEST(s) to get the set of products desired. - sites that want to use the products that will be available in the reconstituted NIMAGE feed should immediately install and switch to using the most recent releases of their Unidata-supplied display and analysis applications This is especially true for GEMPAK users as imagery in netCDF4 format was not supported in Unidata GEMPAK prior to the current release. - the data volume that will be available in the reconstituted NIMAGE feed is actually 20% less than the volume of the sum of the products being delivered in NOAAPort and distributed in the NOTHER and HDS datastreams The reason for this is twofold: - NOAAPort broadcast headers and footers have been removed resulting in a minor reduction of volume - netCDF4 compression has been implemented/implemented better in the products Recap: The reconstituted IDD NIMAGE datastream will contain: - full image scenes that have been created by stitching together the ABI tiles sent in the NOAAPort SBN and distributed in the IDD NOTHER feed - all other Level 2 products being distributed in the NOAAPort SBN in the IDD NOTHER and HDS feeds - value added GLM image/grid products created by Eric Bruning of Texas Tech University - all products are in netCDF-4 format - all products are in the GOES-R mission standard Fixed Grid projection - the names for all products (contained in the LDM/IDD Product IDs) follow the GOES-R mission standard naming convention as detailed in: GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG) https://www.goes-r.gov/users/docs/PUG-GRB-vol4.pdf In all cases, the LDM/IDD Product IDs will consist of fully qualified path names for products, and the file name portion of the fully qualified names follow GOES-R mission standards for naming. The NIMAGE web page referenced above lists example Product IDs for each product. Please send any/all questions/comments about the upcoming change to the IDD NIMAGE feed to: support-datastream@unidata.ucar.edu ```

akrherz commented 5 years ago

updated message from Unidata indicates the new products would be made available on 19 June. I am stuck at the moment with bandwidth/latency issues in my machine room, so hopefully with that cleared later this week, we can take a stab at this.