Remove DWQUPDESDischargers from Internal/OpenSGID

stdavis commented 3 months ago

Summary

Internal Dataset Name: SGID.ENVIRONMENT.DWQUPDESDischargers
External Dataset Name: Utah DWQ UPDES Dischargers
ArcGIS Online Url: https://www.arcgis.com/home/item.html?id=b3eb26c4342f436e96df06a4364b071d

This is part of a group of layers (used in the DEQ Interactive Map) that DWQ has recently moved from File Geodatabases on a network share to AGOL hosted feature layers. This means that we no longer need to pull data from their file share for the interactive map and can instead reference their AGOL items directly. These layer are also currently in SGID. We can now remove them from internal/OpenSGID and our AGOL org and replace them with shared items from DEQ in SGID for ArcGIS.

Migration Guide

This dataset has been replaced by this DEQ AGOL Service.

Action items

These items should be addressed before any further actions are taken.

[x] Should the new DEQ AGOL item be included in SGID on ArcGIS? Yes via shared items from DEQ
[x] Should the new DEQ AGOL item be included in the SGID Index? Yes
[x] Work with Ryan Parker to get his AGOL item standardized and shared to our Hub (@gregbunce, completed: 2024/90/19)

Soft Delete

The purpose of the soft delete is to ensure that all of our users and applications have gracefully migrated off of the dataset. Soft deletes will remain in effect for 14 days. During this time, we will have the ability to restore the dataset to its original SGID offering(s). After these 14 days, the item is then ready for a hard delete.

Note: If this dataset is being replaced, then wait until the new data is publicly available before completing these steps:

[x] Append "(Mature Support)" to the end of the item title in the SGID.META.AGOLItems table (@stdavis, completed: 2024/09/30)
[x] Remove all tags other than "Deprecated" in the SDE metadata (@stdavis, completed: 2024/09/30)
[x] Add note to SDE description noting when layer will be deleted and any replacement layer (@stdavis, completed: 2024/09/30)
[x] Update the SGID Index to point to the DEQ AGOL item (@stdavis, completed: 2024/09/30)
[x] Change Authoritative field to d in SGID.META.AGOLItems to automatically set the Deprecated AGOL flag. Allow the d to persist through one run of Auditor - currently, Auditor runs daily at 5:00am (@stdavis, completed: 2024/09/30)
- After at least one successful run of Auditor:
- [x] Remove the row from the SGID.META.AGOLItems table. This will trigger the removal of this item in Open SGID (@stdavis, completed: 2024/10/14)
- [x] Unshare the item from its SGID group, which will remove it from the SGID on ArcGIS (@stdavis, completed: 2024/10/14) ~- [ ] Check if this layer in the base maps. If so, work with Zach to develop a path forward before you proceed with a Hard Delete (name, completed: 2024/00/00).~

Hard Delete

Target Date: 2024/10/15

Hard deletes are final. It is recommended to complete the soft delete process before moving on to these steps. If you decide to skip the soft delete, note that you will need to incorporate some of those steps here.

[ ] Manually remove data from the Internal SGID (@gregbunce, completed: 2024/00/00)
- [ ] Deprecated database layers should be backed up on Google Drive > AGRC Projects > SGID > deprecated layers > internal_db_sgid_layers (@gregbunce, completed: 2024/00/00).
[ ] Remove ArcGIS Online item: (@gregbunce, completed: 2024/00/00)
- Manually delete the AGOL item
- -OR- ~- Unshare by changing the AGOL_ITEM_ID field in SGID.META.AGOLItems to something other than an Item ID and manually changing the sharing settings (remove the item from all SGID groups, remove Public sharing),~
- -OR- ~- Shelve the data by copying the row from SGID.META.AGOLItems to the AGOLItems_shelved table in AGOL and changing the AGOL_ITEM_ID field in SGID.META.AGOLItems to shelved or some other note.~
- Deprecated AGOL items can be backed up on Google Drive > AGRC Projects > SGID > deprecated layers > agol_sgid_layers. ~- [ ] Remove Farm from AGOL connection (@stdavis, completed: 2024/00/00)~ ~- [ ] Update the SGID Index fields: "arcGisOnline", "openSgid", "openSgidTableName" (@stdavis, completed: 2024/00/00)~
[x] Update relevant gis.utah.gov data pages (including downloadMetadata.ts) (@stdavis, completed: 2024/10/21) ~- [ ] Update SGID.META.AGOLItems table (name, completed: 2024/00/00)~
- cut and paste row to AGOLItems_shelved table if shelving (see below)
- set the AGOL_ITEM_ID field to hosted by <agency> for Farm from AGOL
- set the AGOL_ITEM_ID field to exclude from AGOL to not publish to ArcGIS Online
[ ] Delete Google Drive data (@gregbunce, completed: 2024/00/00)
[x] Remove row from SGID.META.ChangeDetection (@stdavis, completed: 2024/10/21)
[x] Remove data from forklift hashing and receiving (@stdavis, completed: 2024/10/21)
[x] Remove row from data/hashed/changedetection.gdb/TableHashes (@stdavis, completed: 2024/10/21)

Shelve/Static

Choose one based on situation.

~- [ ] Upload to UtahAGRC/AGRC_Shelved folder in AGOL (New shelved item not already in AGOL) (name, completed: 2024/00/00)~ ~- [ ] Move existing AGOL item to AGRC_Shelved AGOL folder (shelving an item already in AGOL) (name, completed: 2024/00/00)~ ~- [ ] Upload to appropriate UtahAGRC/{SGID Category} folder in AGOL (for static datasets) (name, completed: 2024/00/00)~

Add record to table.

~- [ ] Add record to AGOLItems_shelved table in ArcGIS Online with shelved or static in the CATEGORY field (name, completed: 2024/00/00)~

:robot: Automation validation

Assign yourself or someone to check the item by replacing name with their github @name.
Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
~Strike~ out all items that do not apply.

[x] Remove data from Open SGID (@stdavis on 2024/10/21)
~[ ] sgid-index (@steveoh on 2024/00/00)~
[x] Auditor sets appropriate shelved/static/Deprecated information (@jacobdadams on 2024/10/01)

Are there service dependencies

[x] Are clients querying this data with the WebAPI? No, I checked the dashboard and couldn't find any requests.
[x] Notify Don Jackson (don.jackson1 at motorolasolutions.com) if the data is in the Next-Generation 911 Motorola Aware Map? (@gregbunce on 2024/09/16)
[x] Does any other application depend on this data?
Enviro, but I've already pointed it at the new AGOL items. There may be some cleanup in forklift at some point but I don't think that it should block this porter item.

Notification

[x] Twitter (@steveoh)

Group Task Assignments

Check [x] the box when you have assigned all the tasks relevant to your group.

[x] Data Team (@gregbunce)
[x] Dev Team (@steveoh)
[x] Cadastre Team (@rkelson)

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 0 out of 3 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 0 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

steveoh commented 2 months ago

dev team tasks are assigned. I believe the rest of the unassigned are for the data team. let me know if you agree @gregbunce

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 0 out of 3 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 0 out of 3 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 0 out of 3 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

gregbunce commented 2 months ago

note: i'm working with Ryan to get these 7 water-related issues standardized and wired up to hub.

gregbunce commented 2 months ago

okay, we're back in business. the deq layers are wired up to our Hub site. we're ready to proceed.

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 2 months ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
@stdavis has completed 0 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
@jacobdadams has completed 0 out of 1 tasks	:no_entry:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 2 months ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 1 month ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
@stdavis has completed 5 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
jacobdadams has completed 1 out of 1 tasks	:+1:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 1 month ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 1 month ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
@stdavis has completed 5 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
jacobdadams has completed 1 out of 1 tasks	:+1:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 1 month ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:no_entry:
open sgid	:no_entry:
open data	:no_entry:
arcgis online	:no_entry:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 1 month ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
@stdavis has completed 7 out of 12 tasks	:no_entry:
@steveoh has completed 2 out of 3 tasks	:no_entry:
jacobdadams has completed 1 out of 1 tasks	:+1:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 1 month ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 1 month ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
stdavis has completed 12 out of 12 tasks	:+1:
@steveoh has completed 2 out of 3 tasks	:no_entry:
jacobdadams has completed 1 out of 1 tasks	:+1:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 1 month ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 3 weeks ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
stdavis has completed 12 out of 12 tasks	:+1:
jacobdadams has completed 1 out of 1 tasks	:+1:
steveoh has completed 2 out of 2 tasks	:+1:
@rkelson has completed 0 out of 1 tasks	:no_entry:

agrc-conductor commented 3 weeks ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 2 weeks ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
stdavis has completed 12 out of 12 tasks	:+1:
jacobdadams has completed 1 out of 1 tasks	:+1:
steveoh has completed 2 out of 2 tasks	:+1:
rkelson has completed 1 out of 1 tasks	:+1:

agrc-conductor commented 2 weeks ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 1 week ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
stdavis has completed 12 out of 12 tasks	:+1:
jacobdadams has completed 1 out of 1 tasks	:+1:
steveoh has completed 2 out of 2 tasks	:+1:
rkelson has completed 1 out of 1 tasks	:+1:

agrc-conductor commented 1 week ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc-conductor commented 5 days ago

conductor results for tasks - 346

check	status
@gregbunce has completed 3 out of 7 tasks	:no_entry:
stdavis has completed 12 out of 12 tasks	:+1:
jacobdadams has completed 1 out of 1 tasks	:+1:
steveoh has completed 2 out of 2 tasks	:+1:
rkelson has completed 1 out of 1 tasks	:+1:

agrc-conductor commented 5 days ago

conductor results for environment.dwqupdesdischargers

check	status
internal sgid	:no_entry:
meta table	:+1:
stewardship	:no_entry: Did not find environment.dwqupdesdischargers in the worksheet

agrc / porter