search

kcigeospatial / MDOT-SHA-NPDES-Next-Gen

Code and issues related to the MDOT SHA NPDES Project. Project codes: Config = 31, Management = 32.
0 stars 0 forks source link

"Date Abandoned" being populated with "0" after editing/validate run #260

Open KCI-Ablowers opened 6 years ago

KCI-Ablowers commented 6 years ago

For some reason, when a SWMFAC feature is edited, the "Date Abandoned" field is being populated with a "0" after the validate process is run. The editor is listed as "FIELD". I believe the topic of the "FIELD" user has come up before. I am attaching a copy of the features the query widget caught with this error. At first, it seemed like a "one-off" issue but after the most recent "validate" run, it caught it again with the three SWMFAC's I edited.
features (7).zip

johnshiu commented 6 years ago

Is it possible that Collector or the AGOL Web Map is setting a blank value to zero? I am not updating the Date Abandoned value in the code. To test this, you can try editing a SWMFAC, and then closing the map and returning to it to see what the value is set to (forcing it to refresh from the feature service).

Also, whenever the ETL scripts make edits, it will show the last edit user as 'FIELD'. This username refers to the database connection username that makes the update. Therefore, every feature that is processed during a run would have its last editor set to 'FIELD', which is the expected behavior.

KCI-Ablowers commented 6 years ago

I just tested this by-

Not sure what is causing this issue. I thought it may be the offline sync via ArcMap, but I only edited #120303 with that method.

johnshiu commented 6 years ago

Hmm, I"m not sure either. I did notice that in the attached features.csv, there were zeroes in several other columns, Were all of those intended as well? Is it the CSV export function that might export the zeroes out, even if the actual data is empty?

KCI-Ablowers commented 6 years ago

I looked at the feature service and the other zeros are populated there. I think the CSV export is fine. On a side note, @brentreeves75 noticed in the BMP export CSV (from the NDPDES FDM) that a single quotation mark came out like this "15’" in the "Overall Comment" field. I looked for it in the feature service and survey123, both had the single quote present.

It seems like when I cleared the zero out of the "Date abandoned" field that it didn't get repopulated during the next push through the "validate" process. Could it maybe have something to do with the field type?

talllguy commented 6 years ago

This is an integer field operating as a year. Nullable = true for the feature service. Perhaps it is something with the edit template.

talllguy commented 6 years ago

microsoftteams-image

030267 (last edited JSHIU2 on 4/24) has zero written for Date Abandoned. I think I've seen that set to zero on others as well. @KCI-Ablowers Can you verify that the zero returns even after clearing it and running the processing?

talllguy commented 6 years ago

Based on the rule, I think this should appear zero unless it is abandoned.

image

talllguy commented 6 years ago

Per the SOP, these should either contain a four digit date, 9999*, or NULL

image

9999 is only for unknown.

KCI-Ablowers commented 6 years ago

I cannot replicate this for some reason. Steps as follows:

  1. Edit SWMFAC# 030102 after confirming "Date Abandoned" field was blank/null/empty using Collector
  2. Run Validate process in NPDES FDM
  3. Open Edit Web App and run the SWMFAC Query
  4. No ETL Issue flag for date abandoned or date populated with "0" in attribute table
  5. Edit same SWMFAC using the Edit Web App
  6. Save edits and close web app
  7. Run Validate process in NPDES FDM
  8. Open Edit Web App and run the SWMFAC Query
  9. No ETL Issue flag for date abandoned or date populated with "0" in the attribute table
  10. Close web app
  11. Use the offline method to edit the same SWMFAC and sync edits back to server
  12. Run Validate process in NPDES FDM
  13. Open Edit Web App and run the SWMFAC Query
  14. No ETL Issue flag for date abandoned or date populated with "0" in the attribute table

https://github.com/kcigeospatial/MDOT-SHA-NPDES-Next-Gen/issues/260#issuecomment-407257033

Can you verify that the zero returns even after clearing it and running the processing?

I just cleared the "0" for SWMFAC 120303 and ran the validate. The zero did not return and the error was eliminated.

talllguy commented 6 years ago

I think we can close this based on your testing, Andrew. Let's keep an eye out for zeroes later and see if it comes up again.

talllguy commented 5 years ago

@KCI-Ablowers Can you run ETL 1 when you get a chance and let's check if Conveyance 67520808-1b24-4812-a08c-16999972d908 gets flagged for this.

KCI-Ablowers commented 5 years ago

@KCI-Ablowers Can you run ETL 1 when you get a chance and let's check if Conveyance 67520808-1b24-4812-a08c-16999972d908 gets flagged for this.

Do you mean the validate process?

talllguy commented 5 years ago

Yep. I didn’t want to run it in case there’s work in progress stuff in there.

-- Elliott Plack, Business Analyst Geospatial Solutions • KCI Technologies, Inc.

w: (410) 891-1750 m: (410) 206-6448

📧📲

On Sat, Mar 2, 2019 at 12:42 PM -0500, "Andrew Blowers" notifications@github.com<mailto:notifications@github.com> wrote:

@KCI-Ablowershttps://github.com/KCI-Ablowers Can you run ETL 1 when you get a chance and let's check if Conveyance 67520808-1b24-4812-a08c-16999972d908 gets flagged for this.

Do you mean the validate process?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/kcigeospatial/MDOT-SHA-NPDES-Next-Gen/issues/260#issuecomment-468942433, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABvkKYh5F6c6OUSx-A9M5PHu5_h2mfzlks5vSrftgaJpZM4VY3XX.

talllguy commented 5 years ago

@johnshiu it appears this issue has reemerged with the reconcile version. I ran the validate and it came up with many missing Date Abandoned values. The trouble with this is that the editor tool in the web app doesn't allow editing (nor does it even show the) Date Abandoned field. I can't seem to figure out where these zeros are coming in.

image

talllguy commented 5 years ago

I added a new inlet with no attributes at all, then saved. Then I opened the inlet again, and the date abandoned is still null. So, I'm not sure where the zero could be coming in. image

johnshiu commented 5 years ago

@talllguy, hmm, perhaps this is being populated via Collector instead of the web map UI? Also, if you run the ETL for this version, does that new inlet have a date abandoned and a flag set? It may be the script that's updating it to 0, although I don't think that's the case based on the code I've reviewed.

EDIT: I would imagine that this would happen with any numeric field (default to zero) if it happens to this one. Also, if this persists as a problem, I can have the script overwrite the 0's and set it as null.

talllguy commented 5 years ago

@johnshiu These were entered directly in the web app with no Collector in the mix. I can go into NPDES later and clear out the zeros but I am wondering why they appear. Not saying it is your code though, could be an Esri thing. I wish we could figure it out.

johnshiu commented 5 years ago

@talllguy, is it possible to run the ETL script against this version? Maybe that could be triggering the zeroes? Again, I'm not sure though.