BMP Inspection feature service geometry snapping to SWMFAC

[KCI-Ablowers]

@johnshiu and @talllguy

Would it be possible to snap the BMP INSPECTION point that is generated in the feature service to the BMP Centroid or the center of the SWMFAC polygon? This could happen during the "Validate Field Changes" processing. While reviewing MES' initial inspections, Brent and I realized that they were skipping the "Inspection Location" point in the Survey123 form. While this is not carried through to the production NPDES database, it does help with QAQC in the web map (since the table export is not set up yet).

Since the "Inspection Location" is not set to be required and the threshold will not be met with the devices everyone is using in the field, could this snapping replace it in the eyes of SHA?

[johnshiu]

The ETL could auto-update the location, but then it would defeat the purpose of the location QA/QC check.

Please let me know how this should be handled, and I can make the necessary updates.

[KCI-Ablowers]

Is the location QA/QC check part of the ETL Spatial Issues for the BMP Inspections? What does it check for exactly? I looked through the https://github.com/kcigeospatial/MDOT-SHA-NPDES-Next-Gen/blob/master/NPDES_FinalSQLTests.csv document but couldn't find anything about a point location check.

[johnshiu]

No, the ETL does not check at all for the location of the inspection. The QA/QC check I was referring to was the visual check done by the inspector in the AGOL map.

[KCI-Ablowers]

Understood, sorry about that. Maybe we just abandon the idea of a visual QA/QC since we will be able to export the inspection table (by firm) from the editing web app. Snapping the inspections points would assist in a one off "Search for this SWMFAC and the BMP Inspection point is right there" QAQC, but not an overall QAQC (since this will most likely be done on paper or in excel then sent back to the person editing data). Snapping would almost make it better visually for the editor since the points will not be located at the 0,0 mark or however that location is determined.

[talllguy]

It should be required. I will investigate.

[talllguy]

OK. A minimum accuracy is required but it does not prevent submitting the form. That is by design in case no location services are available. We can, however, pass a default geopoint in the Survey123 URL. To so so, we'd need a lat/long pair for the BMP, accessible via attributes.

If we want to do this, we'd need to add two fields to the field schema for BMP for X and Y in decimal degrees. If that data was present and populated, I'd be able to pass that information to survey.

[KCI-Ablowers]

So in that case, the "Inspection Location" point would appear at the center of the BMP (for example) and would only need to be moved rarely, if ever. All based on that lat/long. I think that would be a good work around.

[talllguy]

Right. Plus, if the user's GPS is working, the form will still attempt to get a better coordinate.

[johnshiu]

Hmm, I would recommend not adding the XY fields to the schema, as that will be another set of coordinates to keep track of. I think the LOE of adding those and back populating them across each version would be higher than just adding the auto-populate option to the ETL script.

[KCI-Ablowers]

In situations where the inspection is submitted away from the actual BMP, locating the facility on the "Inspection Location" map is very difficult unless you enter a Lat/Long manually and save the point in the survey. I've done this for some that I have entered in the office or when we were in route to the next BMP. Would there be any real issue in snapping the point to the BMP Centroid or the center of the SWMFAC? If it can be done in python, @johnshiu can work his magic and have it just update new inspection points.

[johnshiu]

Yes, let's go forward with this for now in the ETL.

[johnshiu]

This has been implemented in the ETL at SHA. Please test and let me know if any issues arise.

To clarify, you should run the validation ETL to trigger this, which will update all BMP inspection points to the SWMFAC centroid.

[talllguy]

I like this approach. I will remove the geopoint display from the form since there will not be a need to ensure the form GPS is accurate. The user can simply run the Validation ETL and it will move their inspections to the proper spot.

One future enhancement to consider @johnshiu is to add a random jitter to the 10,000 digit of each coordinate (0.00001). in Maryland, this would equal about one meter of movement per .00001 change. This way the points would be scattered if there are multiple surveys per BMP, making them easier to identify.

[talllguy]

@johnshiu 🚨 🚨 🚨 🚨 🚨 🚨 🚨 There is a critical issue with this BMP point moving feature!! We tested with KCI_1 and all 25 inspections are moved off the map. Here's all 25 KCI_1 points exported to geojson.

Notice how all are along the north pole 🎅 ? Not good. Can you adjust? It might have something to do with a projection issue.

[johnshiu]

Oh, dear; I will take a look at this. Sounds like a projection issue. At least it is easily reversible by re-running a fixed script.

[talllguy]

John, thanks for taking a look! As you said, luckily no real data loss, just movement.

Also, a thought... I am glad we're preserving the form start and end times in the data because anytime things move or are updated, the editor tracking stats are updated. It is helpful to retain the original dates for future reference this way.

[johnshiu]

This should be fixed now; it was indeed a projection issue, and it is now projecting from the MD State Plane to WGS84 correctly.

Also as a technical note, I had to add a check in the code for decimal degrees > 180; the default ArcPy function projected it to 200+ X values, so I needed to drop them back down to -70 by subtracting 360.

[talllguy]

I validated the fix is working, all inspections are back where they should be.

Was the > 180 projection conversion the main cause?

[johnshiu]

The state plane to WGS84 was the main one, but I had to add the extra 180 check after the projection gave me funky results.

[talllguy]

Interesting. Well, this one is now working as expected so we can close.