search

agrc / porter

UGRC tracks the additions, replacements, and deletions of SGID items (in the broadest sense of add, replace, or delete) through issues in this repository.
https://gis.utah.gov/documentation/policy/
MIT License
2 stars 0 forks source link

Add Reference Posts from UDOT #172

Closed gregbunce closed 2 years ago

gregbunce commented 2 years ago

Summary

This dataset is the latest and greatest from UDOT for Utah mile posts. UDOT maintains this layer in their ArcGIS Portal and has shared the URL with us and has shared it via our open data site.

Because of UDOT's funky naming on this layer, UGRC has created a new Item in our AGOL with the name Utah UDOT Mile Reference Posts and pointed the feature service to UDOT's Portal item. This allows us to name the layer as needed, with UDOT continuing to update the data. The only caveat is that the metadata will not remain in scyn with UDOT's Portal layer. This is not a problem in this case, as UDOT has UGRC create the metadata for this layer.

This layer will replace both the existing mile post and 10th of mile post data we are hosting in SGID (which are based on the LRS measures). This new layer is based on the physical signs in the ground. We will provide a link in the data page to the LRS-based data in case folks are looking for it, but UDOT is recommending that folks use the mile reference post data when possible (the physical signs on the ground).

The data should be available in

1 Check [x] all the areas where you expect the data to show up.

The data is of high quality

~- [ ] Sweeper checks have run and passed (name on 2021/00/00)~

Where is the data source

Choose one.

~- [ ] Internal SGID~

Action items

  1. Assign a person who should complete the task by replacing name with their github @name.
  2. Check [x] the box when the task is completed and add the date of completion.
  3. ~Strike~ out all items that do not apply.

:robot: Automation validation

  1. Assign yourself or someone to check the item by replacing name with their github @name.
  2. Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
  3. ~Strike~ out all items that do not apply.

Notification

Group Task Assignments

  1. Check [x] the box when you have assigned all the tasks relevant to your group.
gregbunce commented 2 years ago

@stdavis this one's a little funky so let me know if you want to chat about any of it. for one, they host the data in the ESRI Portal (not AGOL), so, the URL might look different. Also, they are not able to rename the layer name but i was hoping we could so something fancy and use the name we give it in Internal as the name we expose in Open SGID and Open Data. is that possible? (seems like we can do that for Open SGID, but maybe not for Open Data). any thoughts on a fancy name switch-a-roo (@stdavis @steveoh @jacobdadams )

jacobdadams commented 2 years ago

I'm not Scott, but I'm pretty sure for Open Data we're stuck with whatever name they've given it in their Portal, because it's just a direct reference to that item. My vote would be for keeping the name consistent across SGID products, but that's just me.

gregbunce commented 2 years ago

just dropping this here for everyone's reference. The data we're introducing is Mile Reference Posts (they no longer maintain Mile Points but it can be derived from the data we're introducing using the 'Measure' field. The 'Legend' field contains the Reference Post). I'll make sure this is all in the metadata. image

stdavis commented 2 years ago

I don't believe that it matters what the source layer name is when we are pulling from AGOL into internal. We can rename it to whatever we want. I checked out the source and we should be able to access it fine.

steveoh commented 2 years ago

conductor results for tasks - 172

check status
@stdavis has completed 0 out of 3 tasks :no_entry:
@gregbunce has completed 1 out of 5 tasks :no_entry:
@steveoh has completed 0 out of 4 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
@rkelson has completed 0 out of 1 tasks :no_entry:
stdavis commented 2 years ago

@gregbunce Forklift will not create new feature classes in internal, it will only update existing ones. I un-strikethrough-ed the task to create this. I wasn't sure who to assign it to.

gregbunce commented 2 years ago

Thanks for the clarification, Scott. I understand. I'll get it loaded. Once it exists, forklift will keep it updated, correct?

On Tue, Feb 1, 2022, 2:26 PM Scott Davis @.***> wrote:

@gregbunce https://github.com/gregbunce Forklift will not create new feature classes in internal, it will only update existing ones. I un-strikethrough-ed the task to create this. I wasn't sure who to assign it to.

— Reply to this email directly, view it on GitHub https://github.com/agrc/porter/issues/172#issuecomment-1027303837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEA7QTAPCKDTKL2M4LFHYD3UZBFYZANCNFSM5NB3LCZA . You are receiving this because you were mentioned.Message ID: @.***>

stdavis commented 2 years ago

Once it exists, forklift will keep it updated, correct?

Yes, once that forklift PR is merged.

steveoh commented 2 years ago

I'd like to see what comes of the meeting with UDOT to see what the real story is and what work is required here.

gregbunce commented 2 years ago

@stdavis i added the layer to Internal and created a row in the meta table. i feel like we can move forward with the farm to agol piece while we work out the udot name, right?

note: let me know if you have issues setting up farm to agol. i had to download the feature service as a shapefile so the fields got a little funky. I did my best to match what's in the feature service but i might have missed something. just let me know and i can fix it at the feature class level (in Internal).

image

stdavis commented 2 years ago

Looks great, @gregbunce!

@steveoh Can I get a review on https://github.com/agrc/warehouse/pull/134?

stdavis commented 2 years ago

FYI... https://github.com/agrc/warehouse/pull/134#issuecomment-1028272260

gregbunce commented 2 years ago

steve and i scheduled to meet with udot tomorrow morning to clarify the different datasets.

gregbunce commented 2 years ago

Update: let's hang tight for a little while longer. We are talking to udot about publishing a second service with a more user-friendly name. This renamed service would be hidden in their pubic facing sites and only available to us to consume via open data.

gregbunce commented 2 years ago

update: udot republished the service under a new, more friendly name Utah UDOT Mile Reference Posts. we are working on the details as to how they can share it with us, but keep it hidden in their public sites.

gregbunce commented 2 years ago

this item is no longer blocked. ugrc and udot were able to work out a solution in which ugrc created a new item in agol and pointed it to udot's feature service. this allows us to name the item whatever we want, while the data is fully managed by udot. the only caveat is the metadata. in this workflow, udot's metadata does not carry over so we have to copy it over manually, this if fine in this case b/c udot looks to us to create the metadata.

this is the new ugrc layer that points to the udot's data. https://opendata.gis.utah.gov/datasets/utah::utah-udot-mile-reference-posts/explore

i'll get the data page updated with this data.

gregbunce commented 2 years ago

@stdavis not sure how this will affect the warehouse pr (134), but i updated the meta table. the Internal sgid layer will now be sourced from ugrc's hosted feature service. This is the current configuration (NEW):
image

This is how it used to look (OLD): image

gregbunce commented 2 years ago

here's a draft preview of the ugrc data page, as it would look after we implement and complete this issue. please comment/suggest as you see fit when i mark as 'ready' https://deploy-preview-1977--gis-utah-gov.netlify.app/data/transportation/roads-system/#MilepostPoints

agrc-conductor commented 2 years ago

conductor results for tasks - 172

check status
@gregbunce has completed 4 out of 8 tasks :no_entry:
@stdavis has completed 0 out of 2 tasks :no_entry:
@steveoh has completed 0 out of 4 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
@rkelson has completed 0 out of 1 tasks :no_entry:
steveoh commented 2 years ago

@gregbunce this issue says you added the data to internal and created a meta record in early February but there is no open sgid item in transportation. Do I need to debug and figure out why or is something else going on?

gregbunce commented 2 years ago

i'm not aware of anything else going on. i wonder if i added the meta row incorrectly. does that look correct to you?
image

steveoh commented 2 years ago

Other than you are in arcmap, yes it looks right.

gregbunce commented 2 years ago

@stdavis how will we know if the process to update this layer in Internal from the source (AGOL) is working? Do we have to wait for an update at the source to see if it's working?

(the TMI version behind my question) typically, i wouldn't be concerned, but on this layer, i have my reservations. the reason is - this was not a direct load into Internal from the source. i had to download it from UDOT as a shapefile (that was the only option) and then manually rename the field names to match the original UDOT web service (removing the odd shapefile field naming conventions). long story short, I'm hoping i did it correctly and that all the fields names in Internal match the field names in the web service.
(in retrospect i should have asked udot for a fgdb export from their portal)

stdavis commented 2 years ago

It processed successfully in forklift yesterday:

image

The best test is to wait for a change and make sure that it makes it through to internal and open data.

gregbunce commented 2 years ago

great, thanks for the info, scott.

i reached out to udot to see when a change might be coming to this layer, and they're guessing late summer.

stdavis commented 2 years ago

I would say that a successful forklift report is enough to move forward on a porter issue.

steveoh commented 2 years ago

@gregbunce cloudb has been trying to import unsuccessfully since the 17th.

This looks like a problem... There are only 3 attributes visible.

image

gregbunce commented 2 years ago

thanks for the info. yeah, that's an issue on my end. i'll re-import the layer and get if fixed. thanks!

I should be able to revisit this later this afternoon or at the latest first thing tomorrow morning.

stdavis commented 2 years ago

Looks like the forklift data in hashing was messed up. Not sure how. I deleted it and will let it run again tonight.

gregbunce commented 2 years ago

i have a feeling it's b/c the field names don't match up perfectly. i probably mistyped a few of them. I'm reaching to udot now to see if i can get a fdgb of this layer so i can reload it into Internal. I tried to load the feature service into Internal but it errors out. it's not supported..

gregbunce commented 2 years ago

i was able to get a fresh copy of this layer from udot in fgdb format. i replaced the old (shapefile-derived) feature class with this new feature class. fingers crossed. i think this will solve our issues.

gregbunce commented 2 years ago

@stdavis it looks like the update process wiped out the fields again (the geometry seems fine). it should have about 21 fields, but it only has the required two. should i re-load a fresh copy of the data into Intneral?

image

stdavis commented 2 years ago

@gregbunce I believe that I have the data fixed in internal. It seems to be an issue with the older version of Pro on the forklift machine. I tested it on the new forklift machine in GCP and it worked fine. It also worked fine on my dev machine.

@steveoh and I thought that this would be a good excuse to migrate to the new GCP forklift machine. Its first run will be tonight. 🤞

gregbunce commented 2 years ago

Thanks, Scott. I'll hang tight and check it tomorrow.

gregbunce commented 2 years ago

@stdavis your magic worked! aka: the fields are back in the Internal db. everything looks good. thanks for your persistence on this.

steveoh commented 2 years ago

the cloudb service is still having issues that i'm looking into

gregbunce commented 2 years ago

sneaky, hidden carriage returns in the meta table. thanks for catching that steve. that one is on me.

agrc-conductor commented 2 years ago

conductor results for tasks - 172

check status
@gregbunce has completed 6 out of 9 tasks :no_entry:
@stdavis has completed 1 out of 2 tasks :no_entry:
@steveoh has completed 2 out of 4 tasks :no_entry:
@jacobdadams has completed 0 out of 1 tasks :no_entry:
rkelson has completed 1 out of 1 tasks :+1:
agrc-conductor commented 2 years ago

conductor results for transportation.UDOT_MileReferencePosts

check status
internal sgid :+1:
meta table
- item id :+1:
- item name :+1:
- geometry type :+1:
open sgid :+1:
open data :+1:
arcgis online :no_entry:
stewardship
- Description :+1:
- Data Source :+1:
- Website URL :+1:
- Endpoint :+1:
- Data Type :+1:
gregbunce commented 2 years ago

i think we're ready to close this issue out. @steveoh and @jacobdadams it looks like you have the final checkboxes in the Automation validation section.
the last one to check the boxes, feel free to close it out! thanks!

jacobdadams commented 2 years ago

I've verified auditor 👍

steveoh commented 2 years ago

open sgid and the sgid index looks good. what's up with the red slash for arcgis online?

We have Hosted by UGRC via UDOT in the item id. There is an item id of 1680c84cd7fb426194505eec5fead902.

Does this have something to do with forklift @stdavis?

steveoh commented 2 years ago

I think this is good. welcome to the sgid.