Watershed reports with missing station information

[lbross]

How shall we handle the creation of watershed reports if there is no corresponding record for the pourpoint in the AOI Master FCST List? At this time, NWCC is only requiring reports for active forecast stations, but this could change. Multiple use cases could lead to this condition:

1. The pourpoint is too far away from a forecast station. In this case there will likely be no station triplet to serve as a key to the master list. Not sure if the pourpoint will have a USGS station name.
2. The pourpoint has a station triplet but there is no corresponding record in the Master FCST List

We do want to continue generating the data layers if the pourpoint can't be found in the Master FCST list. Do we want to continue trying to generate the watershed report if the pourpoint is not found? The watershed report tool uses the Master FCST list to look up the nwccName, winterStartMonth, winterEndMonth, and huc2 values.

We could substitute for nwccName using the stationName from the pourpoint. NwccName is used for the titles throughout the report. We could set defaults for winterStartMonth and winterEndMonth that are used if no values are retrieved from the master list. I can't think of a substitute for huc2. This could be a problem for Alaska basins because this is what we use to determine if we want to use the Alaska layers.

We have discussed querying a second Master service that includes pourpoints that aren't forecast points. If we do this, we will need to identify that the pourpoint is not a forecast point appropriately in the watershed report. Another option would be to let the analyst set these required parameters in a configuration file at the basin-level. Maybe the analysis.xml? If so, do we need a UI for this or can the analyst edit the XML?

Batch tool interface display: If a report cannot be generated, the grid will display 'failed'. If a report is generated, but with errors, it will display 'errors'.

[jdduh]

Some background information.

1. When AOI was created in BAGIS (V3), the pourpoint and aoi_v don't have the attribute fields (stationTriplet and stationName) that are needed for the BAGIS Pro batch tool (note: the station data in aoi_v is needed only for data management purposes, not directly used in the batch tool)
2. BAGIS Pro has two tools, batch tool and update station data tool, that can populate the station data in the pourpoint and aoi_v.
3. Information on the two "master" lists is documented in https://github.com/PSU-CSAR/awdb-retrieve/issues/5. We need information on how AWDB manages the source data (update frequency and the dependency on USGS station data). In short, the "live" webservices to be published on NRCS AGOL are updated by routinely querying AWDB services through its API. Active forecast stations are stations that have an end date (endDate) set to a future time (e.g., 01/01/2100). There is no separate database table for active forecast stations, so, in theory, there is only one master list on AWDB. Although, NWCC is interested in having a separate active FCSH master list on AGOL.
4. There are several stationTriplet and stationName related fields in the pourpoint/aoi_v featureclasses. For example, "name", "fcstName", "nwccname", "usgsname" and "awdb_id", "usgs_id". stationTriplete and fcstName are considered as the NWCC official data, which should be the source if BAGIS needs these data. nwccname is a BAGIS program-specific attribute. It's value should be identical to fcstName.

The Basin Analysis reports have two major purposes: 1. They are the contents for the watershed characteristics report portal (https://www.wcc.nrcs.usda.gov/ftpref/support/aoi/). Only active forecast station reports are needed. 2. They are background information for any watershed that water forecasters/water managers/hydrologists are interested in. These reports are likely to be requested on-demand and individually. Using the master FCSH list is sufficient for 1, but not for 2 if stationTriplet and stationName are required fields for generating the report. For arbitrarily created AOI (which can be done in BAGIS V3 using just a shapefile), stationTriplet and stationName data might not exist.

We need to have a rule for determining the default values for stationTriplet ("No specified"), stationName (or just fcstName) ("Not specified"), winter_start_month (11), and winter_end_month (3) if they are not available so that the batch tool can still generates reports for these arbitrary AOIs.

One potential modification of the batch/update station data tools is to ask the user to run both tool sequentially so that the station data are only updated in the update station data tool, but not in the batch tool. The batch tool will not generate any report for AOIs without the stationTriplet and stationName data (results in a "failed"). Users must run the update station data tool first to assign stationTriplet and stationName data to the AOI. They must check the tool output log(list) to see if there is any unmatched AOIs. They need to assign stationTriplet ("Not specified" or whatever) and stationName ("Not specified" or whatever) manually to have the reports generated by the batch tool.

[lbross]

The properties that come from the forecast master list are not saved anywhere (nwccName, winter_start_month, winter_end_month, huc2). They are used for the duration of creating the report. It sounds like we may want to add a field to the pourpoint file called fcstName. (Let's start using the label fcstName instead of stationName where appropriate). This field will be populated from the forecast master list and it will be saved. This value can be edited if there is no entry on the forecast master list for this basin. BAGIS-PRO will use this field where it is currently using nwccName. When updating aoi_v we will start adding/editing fcstName instead of stationName. In the tools, these values are copied from the pourpoint file. I like the idea of managing the stationTriplet and fcstName in one tool. In addition to the batch report tool, we also set it when an AOI is selected, if the data is missing, so that will need to be pulled out. Do we need a tool that allows the user to update the stationTriplet and fcst name for stationTriplets that aren't found in the master forecast list? Of can we ask them to update the attributes in the feature classes through ArcGIS Pro? If the update station tool will be more heavily used, does it make sense to put it into its own button/form? Or is it OK being combined with the batch report tool? The defaults for winter start/end months are fine. I will probably add these to the batch tool properties. We need an answer on the HUC2 value for Alaska basins that aren't forecast points.

[jdduh]

I have been updating the station data in ArcGIS Pro manually. There is no need for a tool.

Set default HUC2 to something other than 19 (Alaska), e.g., 00 (or 0).

Perhaps moving the "Update Forecast Station Data" button and its associated check boxes above the AOI list in the GUI. Rename the "Run" button text to "Generate Report." Here is a quick mockup.

"I like the idea of managing the stationTriplet and fcstName in one tool. In addition to the batch report tool, we also set it when an AOI is selected, if the data is missing, so that will need to be pulled out." Can we add a new status message?

• Success: If a report is generated without errors
• Failed: If a report cannot be generated, the grid will display 'failed'
• Errors: If a report is generated, but with errors, it will display 'errors'
• Not Ready: If an AOI doesn't have the required station data. Check station data first and abort any subsequent processing if the station data is missing.

I'm not so sure about using fcstName as the stationName. Not all stations are forecast stations. We need to have a more in-depth discussion on https://github.com/PSU-CSAR/awdb-retrieve/issues/5, specifically related to the attribute values that you mentioned above. Every time when the ACTIVE_FCSH_Station service is being updated, we need a way to transfer the old values to the new service and set default values to those stations that are new.

[lbross]

Part of the problem is that we need to store some station data in the feature class, so that it can be manually updated. But we also want to be able to get the most current information from the ACTIVE_FCSH_Station service. One aside is that the notes on this service say that it will replace the "master" AOI list. With our current design, this service will need to have the winter start and end months and the huc2 to fulfill this requirement. Agreed that fcstName may not be the best column name because not all stations are forecast stations. But we need to have something other than the stationName that can be both manually and automatically updated. Maybe displayName? Here's how it could work for both active forecast stations and ad hoc AOIs (for the stationTriplet and displayName values): Active forecast stations:

1. Run the "update forecast station" tool. ACTIVE_FCSH_Station service is queried to update the stationTriplet and displayName values.
2. Values are updated in pourpoint and aoi_v as appropriate
3. Indicate if values were updated in the log
4. Display status "success" Ad hoc AOIs:
5. Run the "update forecast station" tool. No match is found.
6. Check to see if there are values in stationTriplet and displayName on pourpoint tool. If so, display status "success". Do we also want to add them to aoi_v in this case?
7. If not, display "Not Ready" status in the UI grid
8. Analyst updates the stationTriplet and displayName for 'Not Ready' AOIs. "Not Specified" is an acceptable value for both.
9. Re run the "update forecast station" tool and now all basins should show as "Success".

Do we need to add the huc2 field to the pourpoint feature class? I am fine with using the default 00, but we need to have a way for the analyst to manually indicate that a basin is in Alaska since it seems like some of these will not be forecast points.

I like the suggestions for the UI.

[lbross]

Notes from our meeting on 06-APR-2023

• Modifications will be made to the addIn for v3.x
• stationName and stationTriplet were introduced with BAGIS-Pro. We will continue to use these fields, but manage them only through the Update Forecast station tool
• huc2 will be added as a numeric field to the pourpoint feature class. To be treated as an Alaska AOI, this field must have a value of 19. This field can be edited for Alaska basins that are not forecast points to allow reports. Huc2 won't be copied to aoi_v.
• If no forecast station is found for a basin and the station triplet and station name are null, both fields will be populated with 'Not Specified'. (Update: We will not populate these fields in the feature class, but bagis-pro will be able to handle them being null and use 'Not Specified' in the appropriate places. Leaving them null it is easier to identify AOIs meeting this condition rather than checking for a specific string)
• For the batch report tool, if either stationTriplet or stationName are blank, we will display "Not Ready" in the status and skip the basin
• The in the batch tools window need to be reset before each tool runs (bug)
• Make the comments box shorter when redesigning the gui

[lbross]

I think one thing missing in this spec is updating the stationName and the huc2 if the values in the pourpoint feature class are different from the forecast station web service. Is this a possible use case? Currently if the stationTriplet is found in the forecast station web service and the 'NEAR' tool is not enabled, no updates are made.

Another question: What should the document title be if there is no station triplet. Currently it is "_watershed_report.pdf".

[jdduh]

Users need to set the correct values of stationName and huc2 manually for the AOIs of non-forecast stations. Otherwise, arbitrary default values will be set by BAGIS Pro. How about using "Not specified_watershed_report.pdf." The second not specified AOI will have a report named "Not specified_002_watershed_report.pdf." A related question - how does BAGIS Pro batch tool handle reports with the same name? I have located probably 20+ AOIs that are duplicates of other AOIs (i.e., with the same stationTriplet, but not necessarily the AOI boundaries.)

[lbross]

Just to confirm that we won't update the station name or huc2 for forecast stations that are found in the forecast station web service. If the stationTriplet is the same in the pourpoint feature class and the service, we will leave it alone.

The short answer is that the BAGIS Pro batch tool doesn't handle reports with the same name. You would likely get the report for the last AOI with that station triplet that processed. One would overwrite the next. This will require some thought and work to manage these names for duplicates.

[jdduh]

Let's not worry about the duplicates for now. The good thing is that the batch tool keeps a copy of the report in the AOI folder. We need to talk about the specs for the stations_FCST_ACTIVE webservice when it becomes a priority. At this point, we have a static forecast station master list that BAGIS Pro uses to check if an AOI is associated with an active forecast station.

[lbross]

Proposed UI for batch tools with two tabs. Let me know what you think.

[jdduh]

Looks good. Move the AOI List to the left and disable everything in the Tasks if no AOI is selected.

Add "Switch Selection" or "Select All/None" to the AOI List

Rename Batch Report Tool group heading to "Generate AOI Reports", check the "Copy reports for all AOIs to the maps_publish folder..." as the default. Move the text (The output will be created in the maps_publish folder for each AOI in that folder) in the AOI List tab between the Comments and the check box (see the image below).

There should be a separate Run button for the Forecast Station Data

Rename Snodas Tool Data Export to "aoi_v & pourpoint GeoJson Export", just name the button as "Export"

Add View Log buttons for the Forecast Station Data tool and the GeoJson export tool.

[lbross]

I don't have all of the functions working yet, but please take a look and let me know if the controls/UI are correct.

[jdduh]

Looks good (except for the aoi_v - it became aoiv, v underlined). Can we add online help to the interface for the Forecast Station Data and aoi_V & pourpoint GeoJson Export tools? Similar to the standard ArcGIS Pro geoprocessing tool's online help (see the image below).

The text for the Forecast Station Data is: "Update the stationTriplet, stationName, and HUC data in AOI's aoi_v and pourpoint attribute tables using a published webservice hosted on NRCS ArcGIS Online."

The text for the aoi_v & pourposing GeoJson Export is: "Export AOI's aoi_v and pourpoint featureclasses to GeoJson's multipolygon and point geometry types."

[lbross]

Note that the log button on the current batch tool will likely throw an exception; A casualty of the 2.x to 3.x conversion. It is already fixed on the forthcoming tabbed batch tools. I have all three log buttons working.

I'll take a look at the online help request after I get the rest of the form functions working.

[lbross]

I believe I've made all of the requested changes to the BAGIS-Pro admin tools. There is a new version of the add-in on the basins server. Note that the list of AOIs takes a little longer than it did previously because it is checking the triplet, stationName, and HUC2 values to know if the AOI is "ready". Please let me know if you run into any issues.