Add function to export GeoJson for loading into snodas database

[lbross]

The function should allow the analyst to choose a polygon feature class and its accompanying forecast point feature class. The function should also take the station triplet and basin name as arguments. The station triplet serves as the id in the snodas database. Example: 06050000:MT:USGS. This function will export the GeoJson from the feature classes and create a json payload that can be consumed by the django-snodas loading script.

@jkeifer

[lbross]

@jkeifer: These geometries should be projected to WGS_1984 when exporting, right?

[jkeifer]

Yeah, EPSG:4326 is required for geojson.

On Tue, Aug 9, 2022 at 11:09 Lesley @.***> wrote:

@jkeifer https://github.com/jkeifer: These geometries should be projected to WGS_1984 when exporting, right?

— Reply to this email directly, view it on GitHub https://github.com/PSU-CSAR/bagis-pro/issues/37#issuecomment-1209709597, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKT4Z26V5FL4MMM6EKE3WTVYKNEZANCNFSM556TDXJA . You are receiving this because you were mentioned.Message ID: @.***>

[lbross]

Attached is a sample pourpoint geoJson file generated from ArcGIS Pro, if you want to take a look @jkeifer. Change the file extension to geojson. It loads on my sample django-snodas instance. Still a long way to go for GUI and parameterization, but it's a start!

snodas_output.txt

[lbross]

@jkeifer What are your thoughts on formatted vs unformatted geojson? I saw that the geometries in the GitHub repo are formatted. Here's an excerpt from ESRI:

If you want the JSON representation in the output file to be more readable, you can enable formatting. The output file will be formatted with spaces, tabs, and carriage returns to improve readability. A formatted JSON may be beneficial for application development and testing. However, it is not recommended for production applications since all of the whitespace is unnecessary and ignored by JSON parsers. Additionally, formatted JSON can be significantly larger than its JSON equivalent, and the file size will be larger than its corresponding JSON representation. This can affect application performance.

[lbross]

@jdduh Do you want to support both feature classes and shapefiles? It's doable, but a bit more work.

[jdduh]

Let's not implement the shapefile/featureclass upload option for now. We can wait until we are ready to release the "add AOI" function to NWCC admin users.

[lbross]

I'm confused. I thought I was supposed to be working on an ArcGIS Pro tool that would export the geojson for a point and polygon layer associated with a particular basin so that new AOIs could be imported into the snodas tool. @jkeifer has written a shell script that will import geojson files into the snodas database that works great. This ArcGIS Pro tool is close to being complete. My question was if it needed to support shapefiles in addition to fgdb feature classes. If the answer to this is no, then the tool should be done by our next meeting.

[jkeifer]

@lbross sorry to not reply sooner.

We are not trying to use these files where inefficient parsing or bandwidth usage would be a concern, at least to me. Much more important is the ability to show a readable diff when something changes in the pour point repo, which is enabled by storing the files formatted.

For cases where we are not going to be adding the resulting output json to the pour point repo, unformatted is fine, if less user-readable. If formatting the output from your tool is a challenge, then I would say it not to worry about it, and we can always use a separate tool to do so.

For example, jq is great for this. Or, if jq is unavailable, a Python one-liner can do it too; I think something like this would work:

<unformatted.json python -c 'import sys, json; json.dump(json.load(sys.stdin), sys.stdout, indent=4)' >formatted.json

In any case, I took a look at the sample you provided and it looked good to me.

[lbross]

Thanks @jkeifer. It's super easy to enable the formatting. Just a switch in the Geoprocessing tool. We will move forward with formatted output. I think I just finished the tool.

[lbross]

This is an update on the batch GeoJson export tool. It shares a form with the batch PDF report export tool. I have duplicated the functionality for looking up and adding the awdb_id, stationTriplet, and stationName if they are missing from the pourpoint file. One point of clarification is that the stationName is the USGS stationName not the nwcc name. This is consistent with how this field was used in BAGIS V3. BAGIS-Pro loads the nwcc name when the AOI is set and uses it as specified in the PDF report. I believe @jkeifer was going to add a process when importing the basin to look up the nwcc name from the AOI master list for the name field so this shouldn't be a problem. The "name" that BAGIS-Pro sends will be overwritten. But the stationName will be in the JSON that is sent which might help with troubleshooting. The version of the batch GeoJson export tool that is currently in use has the code to buffer the polygon to avoid multiple polygons. I updated this today so that it works the same as the single AOI export tool. These changes will be included in the next release of the AddIn. Documentation note: The .geojson files are exported to a snodas_geojson subfolder immediately under the basin/folder that is selected at the top of the form. A message to this effect appears when the process completes. The log (snodas_log.txt) is written in the same folder.

[lbross]

Update function that modifies geojson for polygons to meet geojson standard.

[lbross]

Some updates/corrections to my earlier comments:

1. This tool should not look up and add the awdb_id, stationTriplet, and stationName if they are missing from the pourpoint file. This data is managed by the Update Forecast Station data tool.
2. Station name is either the nwccName from the Master Forecast list or "Not Specified". BAGIS V3 never used the stationName field. BAGIS-Pro queries the station name from the pourpoint file when the AOI is set and uses it as specified in the PDF report.

Question: What do we do if someone tries to run the Export for Snodas tool if the stationTriplet is null or "Not Specified"? Should we restrict this tool to valid forecast points only?

[jdduh]

Does the Export for Snodas tool upload the geojson files to the server? We could allow the geojson files to be generated when the station data is missing, but won't allow uploading the geojson files to the server (i.e., SNODAS Tool).

Just notice that the master FCST list doesn't have the "stationName" field, we will need to find a common field name to store the station name to prevent any confusion in the future. We can use the information below to make a decision. Please point out if there is any error.

1. AWDB uses "name"
2. master AOI and FCST lists use "nwccName", but also have "name" and "usgsName"
3. master FCST list has "fcstName" (this might be a one-time naming convention provided by NWCC)
4. AOI pourpoint and aoi_v attribute tables have "stationName"
5. BAGIS Pro update station data tool uses "nwccName" in the master list to update the AOI layers' attribute table (i.e., stationName)
6. BAGIS Pro export geojson tool uses "stationName" in the AOI's pourpoint attribute table to set the "name" value of the pourpoint and AOI polygon.
7. BAGIS Pro batch tool uses AOI pourpoint's "stationName" as the report title.

AWDB is the source of the station name, but I don't feel like just calling it "Name." We will not use "usgsName" in BAGIS Pro. It seems that we just have to pick between "nwccName" and "stationName".

[lbross]

The Export for Snodas tool does NOT upload the geojson files to the server. @jkeifer has written a script that loads the pourpoints into the Snodas database. For the foreseeable future, the .geojson files will need to be copied to a folder on BASINS so that this script can be run. It's up to you if you want to allow the geojson files to be generated for basins with missing station data. The station triplet is the primary key in the snodas database. I don't see a point to generating these files if they can't be successfully uploaded to the snodas database. If you didn't know the files were missing their stationTriplet, you might try to upload them to the Snodas database and it would fail. As of yesterday afternoon

5. BAGIS Pro update station data tool uses "nwccName" in the master FCST list to update the AOI layers' attribute table (i.e., stationName). If an AOI is not in the master FCST list, it's station name can't be automatically updated. An analyst can edit this attribute data manually. The update station tool will not overwrite the station name of a match isn't found with the FCST list.
6. If the "stationName" is null, the BAGIS Pro batch tool uses 'Not Specified' in the report titles.

AWDB is NOT directly the source of the station name. The master FCST list is the source. AWDB is the source of the BA_SNAME field from BAGIS V3.

[jdduh]

Probably we need additional information on the behavior of SNODAS Tool.

1. Does SNODAS Tool use stationTriplet as the key to link the geometry and the database table? In theory, we can set a random number as the stationTriplet value of non-active-forecast AOIs to trick the system into using the AOI's geometry.
2. Does SNODAS Tool verify the validity of stationTriplet value? Given that we just recently acquired the FCST list, I don't think SNODAS Tool check the validity of stationTriplet value.
3. What's the UI to load a new AOI geometry? If it can only be performed by someone who has local access to the server, then we probably have some control of what geojson files can be uploaded. With the "update station data" tool, we have a better way of knowing which AOI is associated with which active forecast station.

I'm inclined to allow the export geojson tool to export AOIs with whatever stationTriplet values.

The AWDB in my original list refers to the NWCC AWDB database. We have a separate issue (https://github.com/PSU-CSAR/awdb-retrieve/issues/5) to create a feature service using the AWDB data. That service will eventually replace the master FCST list.

[lbross]

I will defer questions 1 and 2 to @jkeifer. Loading a new AOI geometry can only be performed by someone who has local access to the server. One concern is that it may be difficult to review the stationTriplets if there is a long list of .geojson files to be imported. However, the stationTriplet is the source of the .geojson file name so strange stationTriplet values should jump out when reviewing the list of files. We should disallow snodas .geojson files from being created if the stationTriplet is null. This would leave the record without a primary key. But I will allow non-standard stationTriplet values to pass through. If the colons aren't as we expect them to be, I'll the geoJson file name will be the same as the stationtriplet value.

[lbross]

Notes from May 4 meeting:

1. The SNODAS Tool uses stationTriplet as a unique key in the database table? We can set a random number as the stationTriplet value of non-active-forecast AOIs to trick the system into using the AOI's geometry. There are no edits on the stationTriplet value.
2. The SNODAS Tool does not verify the validity of stationTriplet value. However it does need to be unique. If a duplicate station triplet is uploaded, it will overwrite the existing geometries in the database. Each pourpoint is stored as a geojson file in the BAGIS-pourpoints repository. The name of the geojson file is derived from the stationTriplet. Unintentional duplicates should be obvious.
3. BAGIS-PRO has no edits for exporting the geoJson. Basins without a triplet will have the filename NotSpecified.geojson.

Regarding the stationName field: We agreed it would be easiest to retain stationName as the label for this data since that is part of some feature classes attribute table schemas. @jdduh would ultimately like to rename the nwccName to stationName in the master forecast list. This will need to be coordinated with new releases of the AddIns.

[jdduh]

@jkeifer The geojson files of the remaining AOIs are now available at ftp://maps.geog.pdx.edu/snodas_geojson/. There is a separate folder, called Exist_need_to_be_replaced, containing the geojson file of an AOI whose boundaries were delineated incorrectly. The polygon of this AOI needs to be replaced (and all the SNODAS data recalculated).