search

agrc / porter

UGRC tracks the additions, replacements, and deletions of SGID items (in the broadest sense of add, replace, or delete) through issues in this repository.
https://gis.utah.gov/documentation/policy/
MIT License
2 stars 0 forks source link

Add Utah Oil and Gas Fields from OGM replacing existing data in AGOL and Open SGID #85

Closed nathankota closed 3 years ago

nathankota commented 3 years ago

Summary

OGM Utah Oil and Gas Fields are an updated version and are being shared to the AGOL Energy group. This shared feature layer will replace the exiting AGRC version of Oil and Gas Fields in AGOL and will also need to be pulled into the internal SDE for distribution to Open Data.

Triage

We are introducing data

The data was or will be added on 2020/00/00 to the following areas

The data is high quality

The data should propagate automatically to

References to the data need to be manually added to

We have verified the data is live

Notification

steveoh commented 3 years ago

Will this use the same schema as our current data

nathankota commented 3 years ago

Yes, the schemas are the same between our layer and OGM's layer.

steveoh commented 3 years ago

My triage is done.

All references to https://opendata.gis.utah.gov/datasets/utah-oil-gas-fields 7788ca033769450185e426686fcc0e09

will be replaced by

https://opendata.gis.utah.gov/datasets/utahDNR::utah-oil-and-gas-fields 5bb6d8d0ea7440568764dc5a28418be8

agrc-conductor commented 3 years ago

conductor results for tasks - 85

check status
@gregbunce has completed 0 out of 4 tasks :no_entry:
@steveoh has completed 7 out of 8 tasks :no_entry:
@rkelson has completed 0 out of 1 tasks :no_entry:
steveoh)(https://docs.google.com/spreadsheets/d/11ASS7LnxgpnD0jN4utzklREgMf1pcvYjcXcIcESHweQ/edit#gid=1 has completed 1 out of 1 tasks :+1:
@stdavis has completed 0 out of 3 tasks :no_entry:
@ZachBeck has completed 0 out of 2 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
stdavis commented 3 years ago

✔️ Updated AGOLItems table to show hosted by DOGM for this dataset.

gregbunce commented 3 years ago

@eneemann (or whomever else has the magic) can you assist me with getting the correct syntax for running sweeper on an agol and/or open data layer? I tried a few different configurations but I can't seem to get the right syntax. thanks!

seems like I consistently get a 'cannot open table' error message. i tried using item id and the table name with both the agol and open data workspace links.

eneemann commented 3 years ago

@gregbunce After initially thinking a bit, I wasn't sure if Sweeper would work based on our workspace/table name construct. However, if you don't attempt to save a report and provide a blank workspace, it appears to work okay. No errors were noted when using the syntax below:

python -m sweeper sweep duplicates --workspace= --table-name=https://services.arcgis.com/ZzrwjTRez6FJiOq4/arcgis/rest/services/Oil_and_Gas_Fields/FeatureServer/0

python -m sweeper sweep empties --workspace= --table-name=https://services.arcgis.com/ZzrwjTRez6FJiOq4/arcgis/rest/services/Oil_and_Gas_Fields/FeatureServer/0

eneemann commented 3 years ago

I got the service URL by plugging in the AGOL ID that Steve listed above and visiting the item's AGOL Page: http://utah.maps.arcgis.com/home/item.html?id=5bb6d8d0ea7440568764dc5a28418be8

gregbunce commented 3 years ago

@eneemann nice work. Yeah, I guess my issues were supplying a workspace and trying to save a report. the workspace makes sense, but i wonder why we can't output a report. thanks!

eneemann commented 3 years ago

@gregbunce I think it's because the table name is built into the filename of the report, which causes problems when the table name is a URL:

OSError: [Errno 22] Invalid argument: 'c:\\\\Temp\\sweeper_run_20201130_0842\\https://services.arcgis.com/ZzrwjTRez6FJiOq4/arcgis/rest/services/Oil_and_Gas_Fields/FeatureServer/0_DuplicateTest_0.txt'

agrc-conductor commented 3 years ago

conductor results for tasks - 85

check status
@gregbunce has completed 0 out of 4 tasks :no_entry:
@steveoh has completed 7 out of 8 tasks :no_entry:
rkelson has completed 1 out of 1 tasks :+1:
steveoh)(https://docs.google.com/spreadsheets/d/11ASS7LnxgpnD0jN4utzklREgMf1pcvYjcXcIcESHweQ/edit#gid=1 has completed 1 out of 1 tasks :+1:
@stdavis has completed 2 out of 3 tasks :no_entry:
@ZachBeck has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
gregbunce commented 3 years ago

@nathankota how well do you know Michael Van Hatten? is it best to email or set up a chat? I wanted to chat with him about some metadata ideas.

we could suggest some additions and/or we could make those additions to our data page and Internal metadata.

also, regarding domains. looks like they are using county fips number instead of county name (fine enough), and the other suspect field is PROD_FORM. maybe we can get those values defined in the metadata and on the data page.

nathankota commented 3 years ago

@gregbunce I'll ping you about this

gregbunce commented 3 years ago

status update on domains and metadata: i sent Michael Van Hatten an email with a few suggestions for additional metadata/domain content. also, mentioned we would carry that over to the sgid data page for that layer.

gregbunce commented 3 years ago

@ZachBeck i think this can be brought into Internal and the meta table now. the data passed the sweeper checks.

I'll work with Michael Van Hatten to expand the metadata a bit on their agol layer. but, that shouldn't affect Internal, right @stdavis -- when we ingest data into Internal from another agency's hosted feature layer, we don't import the metadata, right?

ZachBeck commented 3 years ago

I guess I don't understand how it will get into Internal, someone set the AGOL_ITEM_ID to hosted by DOGM in the meta table

steveoh commented 3 years ago

@ZachBeck i think this can be brought into Internal and the meta table now.

I guess I don't understand how it will get into Internal, someone set the AGOL_ITEM_ID to hosted by DOGM in the meta table

I think this will be a forklift responsibility.

The AGOL_ITEM_ID is set that way so we do not upload our own version to arcgis online.

gregbunce commented 3 years ago

@stdavis this can be brought into Internal now (see the last three comments) - the data passed our sweeper checks.

gregbunce commented 3 years ago

status update on domains and metadata: Michael Van Hatten replied saying he will work on getting the metadata/domain suggestions we made into their hosted feature service.

stdavis commented 3 years ago

Forklift was already wired up to pull this data into internal. ✔️

steveoh commented 3 years ago

What is the best way to verify that this data is live in open data and internal?

stdavis commented 3 years ago

Maybe compare record counts between the OGM AGOL item and open data/internal?

agrc-conductor commented 3 years ago

conductor results for tasks - 85

check status
@gregbunce has completed 1 out of 4 tasks :no_entry:
@steveoh has completed 7 out of 8 tasks :no_entry:
rkelson has completed 1 out of 1 tasks :+1:
steveoh)(https://docs.google.com/spreadsheets/d/11ASS7LnxgpnD0jN4utzklREgMf1pcvYjcXcIcESHweQ/edit#gid=1 has completed 1 out of 1 tasks :+1:
stdavis has completed 3 out of 3 tasks :+1:
@ZachBeck has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
steveoh commented 3 years ago

I've verified this is in open sgid.

ZachBeck commented 3 years ago

looks like it hasn't made it's way to Internal

steveoh commented 3 years ago

looks like it hasn't made it's way to Internal

How did you verify that?

ZachBeck commented 3 years ago

SGID.ENERGY.DNROilGasFields has the same number of records and what appears to be the same geometry as the AGOL version, it also has an empty ShapeArea field and ShapeLength field as well.

gregbunce commented 3 years ago

I reached out to Michael Van Hatten today and he's working on the metadata now. I think we're close on this one.

nathankota commented 3 years ago

Auditor validation crossed-out. Since this is an externally-hosted layer, Auditor will not touch it. Metadata requirements and domain rules compliance are the final pieces.

gregbunce commented 3 years ago

... still waiting. Michael mentioned he is understaffed. I asked how we can assist. I'll follow up again with him this afternoon.

gregbunce commented 3 years ago

Micheal got me the items i requested for our sgid data page. I'll get that page updated and we can move forward. He is happy with the metadata as-is on their agol page. I asked him to put a link on their agol page to our sgid data page if folks want more info.

gregbunce commented 3 years ago

from my perspective, after this pr is merged, this issue can be closed.