UGS-GIO / wetlands

UGS Wetlands web application that compiles a large data set of Utah wetlands data and displays is on an interactive web map
https://geology.utah.gov/apps/wetlands/index.html
0 stars 0 forks source link

I finally have the landscape data ready for the database! #18

Closed wetlandsDiane closed 2 years ago

wetlandsDiane commented 3 years ago

Hi,

I finally have the landscape layers ready for the database. It consists of five layers, each with nearly identical attributes.

The data can be found here: G:\Shared drives\UGS_Wetlands\Projects\LandscapeData\gis\landscapeAnalysisLayers.gdb\finalLayers'cts\LandscapeData\gis\landscapeAnalysisLayers.gdb\finalLayers'

Martha, if you can't access the layers, let me know and I can move them somewhere else.

A few things of note: I added "_double" at the end of a few field names b/c otherwise GIS was turning those layers into string fields instead of double. That part of the name can probably be deleted off the field name now.

All of the final layers are named with "_final" at the end of the name. That can also probably be removed from layer names.

I have an excel spreadsheet that has additonal information about the fields, including names, data type, and units. Some of the fields are only in one of the layers, but most are in all.

landscapeMetricsForDatabase.xlsx

I will likely be adding a few more fields to some of the layers, but that may be several months off.

marthajensen commented 3 years ago

Hi Diane -

The data location is accessible. Thanks! I'll have a look at the data soon

marthajensen commented 3 years ago

This is for our record only from an email chain between Diane and Martha.

Email from Diane on April 27

I have updated that Google spreadsheet. I just put "see additional spreadsheet" for the popup text field. I attached that spreadsheet below (see next comment in this thread for spreadsheets). The spreadsheet shows which fields should show up in the mapper and what the popup text should look like. I also included information about what scale the fields were present at- most fields are present in all five layers, but a few aren't. Also related to the fields, the last six fields in the attached spreadsheet (all the surface water category ones) aren't yet in the data I sent you. I don't have the data yet, but we could add dummy fields for them for now if you'd like.

Can you guys initially bill UW18 for any work on this project? I am trying to spend out this grant this fiscal year. Once you have billed up to 40 hours on UW18, let me know b/c I will likely have you change over to UW19, though it will depend on where we are at with billing on UW18.

Email from Diane on April 28:

I don't currently plan to have the description field included in the app. I thought perhaps we could use it for tool tips, but first I want to see how crazy the data table will look in the app, since there are so many fields.

I can create an MXD showing how I want the data styled and share that with you. It'll take me a minute b/c currently my work computer isn't working and I don't have GIS on my personal computer! Hopefully this will get resolved soon.

I updated that spreadsheet with a description of how I want plots to show up. I should mention that the plots aren't ready quite yet.

Email from Diane on April 29:

Computer is working again! Woo hoo!

I forgot to mention that I highlighted in yellow on that spreadsheet pop up names that have a squared value (i.e., km2) in it.

I'm working on the map and will put a link to it on GitHub. I suppose this whole email chain should have taken place over GitHub, right? :)

marthajensen commented 3 years ago

Spreadsheets for the popups in the app:

https://docs.google.com/spreadsheets/d/1zDJkZQA3YGEFaqqUdxCgjwvkCI7GL2Y8FJVin9t-OPw/edit#gid=0 metricDescriptionAndPopups (2).xlsx

marthajensen commented 3 years ago

Hi Diane,

I reviewed the fields in the database and added some comments to the following spreadsheet:

metricDescriptionAndPopupsForReview.xlsx

In the spreadsheet, I added two new columns; the first column I added is called "comments." "Comments" contains information on the fields that were changed in all of the feature classes and why there were changed. The second column I added is called "Final field name" - this column matches all of the current field names in my copy of the database. If wanted, I can replace your layers at G:\Shared drives\UGS_Wetlands\Projects\LandscapeData\gis\landscapeAnalysisLayers.gdb\finalLayers'cts with the feature classes that I have modified. Just let me know.

I also changed the "Scales" in this spreadsheet on all of the "surface_water_xxx" fields to All.

My changes are highlighted in red so that you can easily see them. Please get back to me about the questions on Line 31 and 33. Thanks!

wetlandsDiane commented 3 years ago

Hi Martha

Thanks for putting that together!

Yes, please change riparian_percent_area to riparian_percent and change riparian_herbaceous to riparian_herbaceous_area.

Rebecca is working on symbolizing all of the layers right now. I imagine it would be better if she worked with the correct field names rather than the version she is working with. She might have to fix a few of the features she's working with once the names are changed, but I don't think that would be too challenging. So yes, please go ahead and replace our features with the features classes you modified.

Thanks! Diane

wetlandsDiane commented 3 years ago

Actually, Martha, can you reach out to Rebecca and coordinate with her before you make any changes? Just so it doesn't completely take her by surprise in the middle of her work.

Thanks

wetlandsDiane commented 3 years ago

Hi Martha,

I'm trying to pivot to using this instead of email. I'm not sure how to add Rebecca Lee to the project; can you help me with that? I'm not sure if she has a GitHub account.

On the latest draft of the landscape data, there are a lot of null values in the _count fields: riverine_count, lake_count, pond_count, shore_count, woody_count, emergent_count. These are all fields you recently updated from the original values. I checked the original data and there aren't any nulls values in the originals. Can you check what you did and run the update again?

Thanks! Diane

marthajensen commented 3 years ago

@wetlandsDiane I found NULL values in the huc12_final layer - count fields - only. I believe they were a result of a bad join on the string huc 12 key field. I added a new Double field, temporarily, to the huc12_final layer and joined on a Double key instead. All of the NULL values were updated. I checked all the other datasets and don't see any NULL values in any of them (where they shouldn't be). I also visually inspected the values in other columns in both the huc12_final layer as well as the other datasets and they seem right.

Marshall will add Rebecca to Github. Thanks!

wetlandsDiane commented 3 years ago

@jjhi11, @rlee26

After some more data-wrangling, Rebecca has put together a beautiful map showing how all of the fields should be symbolized. She applied the symbology to the HUC12 layer except in cases where the variables are only present at the HUC8 scale. In some cases, the symbology was developed manually and in other cases it was developed using quantiles, b/c in some cases only the quantiles would make sense at all the different spatial scales.

The map can be found here: M:\Shared drives\UGS_Wetlands\Projects\LandscapeData\gis\VisualizeProperties_RL_5.mapx

There is a spreadsheet that lists all of the field names, data types, whether or not a particular field should be symbolized, what type of break was used in the symbology, whether a field should show up in pop-ups, pop-up names, etc. etc. This spreadsheet should supersede previous. table attached below.

I would suggest working on the map with just one of the spatial scales initially and letting us have one more look at the symbology and everything else before trying to bring in any additional layers.

Let me know if you need any more info from me!

metricDescriptionAndPopups_20210517.xlsx

marthajensen commented 3 years ago

@jjhi11 the new wetlands data is ready for you

The data has been published to a new ArcGIS Server service:

https://webmaps.geology.utah.gov/arcgis/rest/services/Wetlands/Wetland_Landscape_Data/MapServer

To build the web popups for the 5 new layers, please use the following two spreadsheets:

Google sheet with basic feature class info: https://docs.google.com/spreadsheets/d/1zDJkZQA3YGEFaqqUdxCgjwvkCI7GL2Y8FJVin9t-OPw/edit#gid=0

The additional spreadsheet referenced in the Google Sheet above: metricDescriptionAndPopups_20210517.xlsx

From Diane (see last comment in this thread): "There is a spreadsheet that lists all of the field names, data types, whether or not a particular field should be symbolized, what type of break was used in the symbology, whether a field should show up in pop-ups, pop-up names, etc. etc. This spreadsheet should supersede previous. table attached below.

I would suggest working on the map with just one of the spatial scales initially and letting us have one more look at the symbology and everything else before trying to bring in any additional layers."

For the symbolization of each layer, please use the following map:

""\168.180.168.181\geology\html\apps\jay\Wetlands_symbolization.mpkx""

wetlandsDiane commented 3 years ago

@marthajensen

Rebecca put together plots for the surface water extent data at the HUC12 level. They are located here for now: M:\Shared drives\UGS_Wetlands\Projects\hydrologySpatialAnalysis\Summary Analysis\StatewideHUC12 These aren't the final plots, but I wanted to share a version of them so that Jay would have them to develop around.

The plots are named by the unique key. There should be one for every HUC12- let me know if that is not the case.

Let men know if you have any questions or need any more info about these or for them to be formatted differently.

Thanks Diane

jjhi11 commented 3 years ago

@jjhi11 the plots have a URL in the surface_water_plot field in the huc12 layer only. This field exists in all the datasets, but we only have information for the huc12 layer currently

marthajensen commented 3 years ago

@wetlandsDiane --

Quick update on this project and the hours I have charged. At this point, I have reformatted the data like we discussed, prepared the data/symbolization for the web app, made URL's for the surface water plots (HUC 12 only), added the plots to our web server, and created a new ArcGIS Server service (called Wetlands_Landscape_Data). Jay is going to start adding the data to the Wetlands application this week. As we work through adding it to the web app, one or both of us might have questions for you this week or next week.

wetlandsDiane commented 3 years ago

Hello, Can you give me an update on this project, including hours billed to UW19 for this fiscal year and hours estimated until completion? I am working on budget stuff for Jodi this week, so that info would be really useful. Also, I am just kind of curious. As a reminder, I need to get something to the EPA by the end of September.

Thanks

jjhi11 commented 3 years ago

@wetlandsDiane I believe it has been about 65 hours. I am close to a solution though so the end of September should be fine.

wetlandsDiane commented 3 years ago

Thanks Jay! Do you have an estimate for the number of additional hours you will need?

On Tue, Aug 10, 2021 at 11:01 AM Jay Hill @.***> wrote:

@wetlandsDiane https://github.com/wetlandsDiane I believe it has been about 65 hours. I am close to a solution though so the end of September should be fine.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/UGS-GIO/wetlands/issues/18#issuecomment-896154092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3KRTWWLCQN4R2MECISPULT4FLODANCNFSM42VYT2MA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Diane Menuz Wetland Program Coordinator Utah Geological Survey 1594 W. North Temple P.O. Box 146100 Salt Lake City, UT 84114 801.538.7436 http://geology.utah.gov/resources/wetlands/

jjhi11 commented 3 years ago

@wetlandsDiane should be around 40 to 60 more hours.

wetlandsDiane commented 2 years ago

@jjhi11

Hi Jay, I did a quick review of the landscape data. It looks really good; I just have minor comments that are hopefully easy to address. See the list below.

The only other feature that I'm hoping to build in at this point is an option for downloading the database and metadata. I'm not sure the best place to put that link. It can obviously be part of the Database Contents text, but it would also be nice for that info to be part of the legend or the box where people select scale and fields. Legend would be ideal b/c then then that info could be next to the study results as well, but I'm not sure how that would be implemented. Maybe best to chat about that feature at a future point?

  1. In the pop-ups, plots only show up for the HUC12 features. There should also be HUC8 plots (though not any other scale).
  2. The HUC12 pop-ups are formatted somewhat different from the other pop-ups.
  3. The fields that show up in the pop-ups include Name and Number. Can we change these to the following For HUC12 scales: Watershed Name Watershed Identifier For HUC8 scales Sub-basin Name Sub-basin Identifier If it is a pain to use different text at different scales, we could instead use HUC Name and HUC Identifier for all scales in the pop-ups
  4. Can you set the default classification to quantile instead of equal interval?
  5. Can you change the order in which the scales show up in the Select Scale box in order of smallest to largest? HUC12 by Ecoregoin, Huc12, HUC8 by Ecoregoin, HUC8, Ecoregion
  6. Can you change the grey “Other” value to say “No data” instead. Or maybe “N/A” I believe this shows up primarily in the riparian fields, ratios fields, and Sen’s slopes and represents times when the results isn’t applicable.
  7. Remove the 30 year surface water trend and Sen’s slope fields from all scales except HUC8 and HUC12. The fields can be removed from the database by Martha or just not show up in the drop down
  8. At the HUC8 scale only, add the fields pct_wells_rising, pct_wells_falling, mean_falling_slope and mean_rising_slope to the drop down menu
  9. The “Growing season 30-year trend” display is very different from the other displays. To make it match a little more, I would suggest updating the outline color to the same color as the outline for the other metrics. We can keep the red, blue and white as is but maybe make the yellowish color slightly darker so we can see the difference between that color and white a little better
jjhi11 commented 2 years ago

@wetlandsDiane these have all been resolved except for #6. I will discuss with @marthajensen about a solution

wetlandsDiane commented 2 years ago

Awesome, thanks!

On Thu, Dec 2, 2021 at 2:08 PM Jay Hill @.***> wrote:

@wetlandsDiane https://github.com/wetlandsDiane these have all been resolved except for #6 https://github.com/UGS-GIO/wetlands/issues/6. I will discuss with @marthajensen https://github.com/marthajensen about a solution

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/UGS-GIO/wetlands/issues/18#issuecomment-985002005, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3KRTUTYPXN4D2UGUMMZVDUO7N4NANCNFSM42VYT2MA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Diane Menuz Wetland Program Coordinator Utah Geological Survey 1594 W. North Temple P.O. Box 146100 Salt Lake City, UT 84114 801.538.7436 http://geology.utah.gov/resources/wetlands/