search

agrc / porter

UGRC tracks the additions, replacements, and deletions of SGID items (in the broadest sense of add, replace, or delete) through issues in this repository.
https://gis.utah.gov/documentation/policy/
MIT License
2 stars 0 forks source link

Add skyshade to raster.utah.gov from Jake's hard drive #324

Open jacobdadams opened 5 months ago

jacobdadams commented 5 months ago

Summary

Several audience members at the 2024 UGIC Iron Cartographer sessions wanted access to our skyshade. This is the same dataset used in Zach's basemaps but converted to 8-bit int and chunked into the same chunks as the source USGS DEMs.

I figured a porter issue is the best way to start the discussion of getting it onto raster.utah.gov.

The data should be available in

1 Check [x] all the areas where you expect the data to show up.

The data is of high quality

Where is the data source

Choose one.

Action items

  1. Assign a person who should complete the task by replacing name with their github @name.
  2. Check [x] the box when the task is completed and add the date of completion.
  3. ~Strike~ out all items that do not apply.

:robot: Automation validation

  1. Assign yourself or someone to check the item by replacing name with their github @name.
  2. Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
  3. ~Strike~ out all items that do not apply.

Notification

Group Task Assignments

  1. Check [x] the box when you have assigned all the tasks relevant to your group.
steveoh commented 5 months ago

Discover or AGOL are the places we have available to serve this sort of data. I assume it would be tiled?

jacobdadams commented 5 months ago

Sorry, I should have explained more. The skyshade is a pure raster product, like a hillshade but with a significantly longer processing time (it took about a week to run on my old office desktop). The output raster needs to be styled to a user's preference, so pre-rendering wouldn't work.

The original output was something like 24gb, but I was able to get it down to about 4gb in the process of dividing it into the same sized chunks we currently serve the underlying 10m DEM in. At this size, a single download wouldn't be too much of a burden on users, so maybe adding it as a file-only AGOL item and linking to it from a datapage would be the better course of action.

steveoh commented 5 months ago

Yeah or GCS is a cheap option for downloads. So, what would be shown in raster.utah.gov?

jacobdadams commented 5 months ago

If we went the raster.utah.gov route, it'd look the same as when you go to download the USGS DEMs- a bunch of tiles that would link to zip files in GCS.

I'm intrigued by the idea of hosting it as an item in AGOL. I'd have to check the storage costs, but then we could expose it through Open Data as well.

steveoh commented 5 months ago

Does that mean that raster would only have a tile index? I use the preview feature but that wouldn't be available would it?

jacobdadams commented 5 months ago

huh, I guess I've never used the preview feature and forgot it was there. Yeah, without a served version of it somewhere, it would be like the Lidar DEMs- just an extent/tile index.

Rick can correct me if I'm wrong, but I don't think it'd take too much space/cost to make a layer on Discover, but I don't know that it's worth it just for the extent. It wouldn't be terribly useful as its own layer... maybe... I mean, I guess I could do a "default" style that I like and then we could have that as a pre-canned product in discover with the raw data available for download if desired.

steveoh commented 5 months ago

I guess I could do a "default" style that I like and then we could have that as a pre-canned product in discover with the raw data available for download if desired.

I like this idea

agrc-conductor commented 4 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 0 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry:
agrc-conductor commented 4 months ago

conductor results for category.internaltablename

check status
internal sgid :no_entry:
meta table
- item id :no_entry:
- item name :no_entry:
- geometry type :no_entry:
stewardship :no_entry: Did not find category.internaltablename in the worksheet
jacobdadams commented 4 months ago

Rick and I chatted about this today and came up with the following plan to move forward:

  1. Make a single, statewide zip available for download through a page on our website. This can be hosted on AGOL for 5 credits a month and also exposed through Hub/open data. The website page will include tips and tricks for using it.
  2. Do an in-place replacement of our Hillshade layer in Discover with a cached tile version of the skyshade. This gets it out there to more people and updates a less-used service (may need to run some numbers to see the actual usage of the hillshade service)
  3. At some point in the future, if needed, add the skyshade zipped tiles (same tiling scheme as the 10m DEM downloads) to raster.utah.gov under a new category in the "Step 1" box (or however it looks when we rebuild the app).

I can start step one right away, and I'll need to work with @stdavis on step two. Step three can wait until later™.

jacobdadams commented 4 months ago

discover service usage courtesy of Rick: image

stdavis commented 4 months ago

So the hillshade would become the preview service for this new layer in raster.utah.gov?

jacobdadams commented 4 months ago

correct, if/when we get there.

agrc-conductor commented 4 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 1 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry:
agrc-conductor commented 4 months ago

conductor results for category.internaltablename

check status
internal sgid :no_entry:
meta table
- item id :no_entry:
- item name :no_entry:
- geometry type :no_entry:
stewardship :no_entry: Did not find category.internaltablename in the worksheet
agrc-conductor commented 4 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 1 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry:
agrc-conductor commented 4 months ago

conductor results for category.internaltablename

check status
internal sgid :no_entry:
meta table
- item id :no_entry:
- item name :no_entry:
- geometry type :no_entry:
stewardship :no_entry: Did not find category.internaltablename in the worksheet
agrc-conductor commented 4 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 1 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry:
agrc-conductor commented 4 months ago

conductor results for category.internaltablename

check status
internal sgid :no_entry:
meta table
- item id :no_entry:
- item name :no_entry:
- geometry type :no_entry:
stewardship :no_entry: Did not find category.internaltablename in the worksheet
agrc-conductor commented 3 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 1 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry:
steveoh commented 3 months ago

Is this a stretch goal or needing a waiting or blocked label?

nathankota commented 3 months ago

Added waiting label to pause Conductor notifications since this is in the FY25 Q1 I&P Sprint 6

agrc-conductor commented 3 months ago

conductor results for tasks - 324

check status
@rkelson has completed 0 out of 2 tasks :no_entry:
@stdavis has completed 0 out of 1 tasks :no_entry:
@jacobdadams has completed 0 out of 2 tasks :no_entry:
@brigsz has completed 0 out of 1 tasks :no_entry:
@steveoh has completed 1 out of 3 tasks :no_entry:
@gregbunce has completed 0 out of 1 tasks :no_entry: