BRAT Tasks: BRAT model runs due 10/8/18 and summary products due 10/15/18

wally-mac commented 6 years ago

@MatthewMeier, @Albonicomt and @CHafen we have some upcoming deadlines for a few BRAT projects that I'd to tell you about (remind you of). Here that describes the tasks in some detail. Mic please decide how much of this you'd like to pass on to Chalese based on your other workload, I know that Sara has assigned you some tasks and that you are working on the Escalante maps.

The projects include: John Day BRAT, NF Brunt River BRAT, and Middle Columbia BRAT. There are milestones for each of these projects. Search milestones under 'Issues' to find them.

Please let me know ASAP if you concerns regarding reaching these deadlines.

Thanks!

wally-mac commented 6 years ago

Update: @Albonicomt, has graciously offered tp run BRAT for the Middle Columbia to help us reach our deadline. Thanks Mic!

wally-mac commented 6 years ago

Update: @banderson1618 and @MatthewMeier are currently working on getting a new release of BRAT completed, so we are in a bit of a "holding pattern". In the meantime, @CHafen and @Albonicomt let's get the new (see #180) base mapping standard apply to the John Day BRAT (all four HUC 8's), NF Brunt River BRAT, and Middle Columbia BRAT. Does this make sense? Mic and Chalese can you please work out who does what? This is a high priority task that will help us reach our deadline of 10/15/18. So let's please get started working on this NOW.

Thanks!

Albonicomt commented 6 years ago

@wally-mac, sounds good. @CHafen and I will coordinate.

MatthewMeier commented 6 years ago

@wally-mac we will get this done and I have been trying to get a hold of @banderson1618 to see what I can help him out with. @Albonicomt I will let you know of the progress on that and if we need to run the data through the past release or not. Just make sure that you have all the inputs and possibly mainchannel edits done. We could do an attribute table join and copy fields later. Thanks @Albonicomt I will give you a call soon regarding your issues you have seen in the inputs for the columbia data.

wally-mac commented 6 years ago

Matt, thank you for the update and commitment to getting this done, I appreciate it.

MatthewMeier commented 6 years ago

@wally-mac @Albonicomt and @CHafen have already gotten started on the base map for the summary products and we are working through some of the issues that @banderson1618 was facing

CHafen commented 6 years ago

@wally-mac are we doing the John Day all together or separating it out into the four HUC 8s?

wally-mac commented 6 years ago

See if you can pull it off as one but show HUC 8 boundaries and show the summaries of each. Does that make sense?

On Fri, Oct 5, 2018 at 1:53 PM CHafen notifications@github.com wrote:

@wally-mac https://github.com/wally-mac are we doing the John Day all together or separating it out into the four HUC 8s?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-427480596, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUlt6ILjfwqN7o6xBH8hOhlhe_51Vks5uh7itgaJpZM4XED_T .

-- Wally Macfarlane 435.512.1839

CHafen commented 6 years ago

Yes, makes sense, will do

MatthewMeier commented 6 years ago

@wally-mac @joewheaton Here is a progress report on the deliverable progress for this project; The inputs and editing for the (John Day and the North Fork Burnt) are completed up til the ihyd step in BRAT. The regression equations are provided in the watersheds metadata txt file that is attached and a threshold determined. AKA they will be good to go for a new run of the BRAT tool and editing will be minimized to a simple attribute table join or calculation using the work @Albonicomt and I have just completed. On a serious and confusing note the Middle Columbia Hood is running into troubles... @Albonicomt and myself couldn't get it past the project builder after using various techniques which Mic will describe in another comment and without @banderson1618 help we are at a loss. We will touch base with @banderson1618 and yourselves on Monday.

Albonicomt commented 6 years ago

@wally-mac, @joewheaton , @bangen and @banderson1618 Having problems running Middle Columbia Hood - 17070105...

I have tried running the Middle Columbia Hood on several versions of BRAT: 3.0.9 ; 3.0.17 ; 3.0.19 ; and the Master PyBRAT.
In BRAT 3.0.19, during "Step 1: BRAT Project Builder," The tools starts for about 5 seconds and then crashes.
In the older versions of BRAT I get the Error code below, during Step1. The tool can not get passed a certain point during step1 and will not even populate the BRAT project folder fully. Here is the Error I get with the older versions of BRAT:

Troubleshoothing:

I thought it might be the versions of ArcGIS, I am running, so I updated from ArcGIS10.5 to ArcGIS10.6.1 and I still run into the same issues.
I have regathered the data from their original sources, made sure all was projected correctly, but still had the same issues. @MatthewMeier, has tried several times as well and he receives the same issues... We are pretty stumped on this one...

Here is a link to the latest processed data: https://usu.box.com/s/m9ealh16b64xsd4gnxisjak7lsuidh1t

Here is a link to the rest of the original data: https://usu.box.com/s/t4pne53e220pf7e6c27iao8264dgbp7i

banderson1618 commented 6 years ago

@Albonicomt, did you ever get this resolved? If not, try the following:

Go to line 178 in BRATProject.py. You should see the following block of code:

        slope_folder = make_folder(dem_folder_path, "Slope")
        slope_file = os.path.join(slope_folder, "Slope.tif")
        out_slope = arcpy.sa.Slope(dem_file)
        out_slope.save(slope_file)
        make_layer(slope_folder, slope_file, "Slope", slope_symbology, is_raster=True)

Place a pound sign/number sign/hashtag in front of each line, so it looks like the following:

    # slope_folder = make_folder(dem_folder_path, "Slope")
    # slope_file = os.path.join(slope_folder, "Slope.tif")
    # out_slope = arcpy.sa.Slope(dem_file)
    # out_slope.save(slope_file)
    # make_layer(slope_folder, slope_file, "Slope", slope_symbology, is_raster=True)

That will comment out the code, making it not run anymore. It won't carry on into any future versions of pyBRAT that you download, but it should work as a hotfix.

It will mean that your tool will no longer create a slope layer to go with your layer package, but that's probably acceptable.

Albonicomt commented 6 years ago

@banderson1618, Sweet Man! I will give it a shot.

MatthewMeier commented 6 years ago

@wally-mac I have gotten the tool to work through the BRAT table and pushing it through the BRAT tool the rest of the way. I also have another running on a vm to get two done at the same time. I will keep you posted on progress

wally-mac commented 6 years ago

That’s good news. Thanks for the update. Keep me posted.

On Tue, Oct 9, 2018 at 4:17 PM Matt Meier notifications@github.com wrote:

@wally-mac https://github.com/wally-mac I have gotten the tool to work through the BRAT table and pushing it through the BRAT tool the rest of the way. I also have another running on a vm to get two done at the same time. I will keep you posted on progress

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-428372451, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUiFJyDxVyQ_11AEXaWAMGn-fTyaCks5ujSCAgaJpZM4XED_T .

-- Sent from Gmail Mobile Sorry for typos and brevity courtesy of my phone.

MatthewMeier commented 6 years ago

@wally-mac I have some questions about the layer package generator output. The following video illustrates this question.https://youtu.be/MMscD2p4dac

wally-mac commented 6 years ago

@MatthewMeier and @Albonicomt can you please work through these questions in Matt's video together? Please keep me posted on your progress here. Thanks

MatthewMeier commented 6 years ago

@wally-mac We will take a look at it.

wally-mac commented 6 years ago

@MatthewMeier here is a short video that @Albonicomt made to describe his method for manipulating the layer packages to include only perennial streams. Mic, is this the same process that you suggest Matt uses? If so, please make this video available here so others can have access to it.

As for the symbology questions. This is the first time that the new management model has been run so you'll need to take a stab at symbology. Matt and Mic to understand how Joe wants this displayed you'll need to read #133 and #175 and watch the associated videos. Please take to time to read them now if you can. Without this background information we'll be unable to understand what Joe's is trying to convey with these layers.

Albonicomt commented 6 years ago

@wally-mac and @MatthewMeier I will take a look at these threads. Thanks for the references. I will shout out with any questions or advice.

MatthewMeier commented 6 years ago

@wally-mac @joewheaton , Up to date runs with the newest version of BRAT can be found here for the John Day. Completed up to final clipping of the layer packages: Upper John Day 17070201 North Fork John Day 17070202 Editing of the Layer package: Middle John Day 17070203 Currently being ran through BRAT(these are links to just file locations not actual finished data products): Lower John Day 17070204 North Fork Burnt 17050202

MatthewMeier commented 6 years ago

Update the Lower John Day 17070204 BRAT is complete and just needs the Layer package clipped.

wally-mac commented 6 years ago

Great! Thanks for the update. It would be super useful to get all the watersheds "clipped" to the perennial extent so that: Chalese and Mic can make the summary produce and Joe and I can have a look at the data in preparation for the workshop. When do you anticipate having this (clipping" to the perennial extent) completed? How do these outputs compare to the original runs? Are the outputs logical and coherent? Please advise and thanks!

MatthewMeier commented 6 years ago

Mic has completed the clipping of the layer package to perennial for the North Fork John day and will up load the location of that soon. He has created the excel pie charts and sent it off to Chalese for map making. As far as truthing the outputs both of us have interrogated the outputs and they are looking accurate to us. I have looked at the differences between the the current and previous run of BRAT for the North John Day and there are some differences in the capacity outputs, which are what is in common with the new and previous outputs. I have interrogated the vegetation layers and looking at the vegcodes and what kind of vegetation they are they look validated to me. There are also differences in where the segments lye which make a difference in the calculations as well. The other outputs from looking at them briefly overall and some segments in depth more and I am not seeing any red flags. Mic has created symbology for the distance from layers in the Human Beaver Conflict. My goals today are to get the NFB through BRAT and about halfway through 17070201 clipping of the layer package but the full network is up on box so if you need to interogate it with Joe and we don't get them all done the full network lpk are under the individual runs in the outputs folder.

wally-mac commented 6 years ago

Thanks for the update.

On Thu, Oct 11, 2018 at 5:16 PM Matt Meier notifications@github.com wrote:

Mic has completed the clipping of the layer package to perennial for the North Fork John day and will up load the location of that soon. He has created the excel pie charts and sent it off to Chalese for map making. As far as truthing the outputs both of us have interrogated the outputs and they are looking accurate to us. I have looked at the differences between the the current and previous run of BRAT for the North John Day and there are some differences in the capacity outputs, which are what is in common with the new and previous outputs. I have interrogated the vegetation layers and looking at the vegcodes and what kind of vegetation they are they look validated to me. There are also differences in where the segments lye which make a difference in the calculations as well. The other outputs from looking at them briefly overall and some segments in depth more and I am not seeing any red flags. Mic has created symbology for the distance from layers in the Human Beaver Conflict. My goals today are to get the NFB through BRAT and about halfway through 17070201 clipping of the layer package but the full network is up on box so if you need to interogate it with Joe and we don't get them all done the full network lpk are under the individual runs in the outputs folder.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-429152923, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUmi4EpFcWFvLYD88Nc1tSPtRtWqFks5uj9FDgaJpZM4XED_T .

-- Sent from Gmail Mobile Sorry for typos and brevity courtesy of my phone.

MatthewMeier commented 6 years ago

Wally, here is an update so that Joe and yourself know what is going on...

BRAT completed and layer package clipped: Upper John Day 17070201 (Chalese is working on the summary products now,Mic created the excel work already)Mic will also share the link to the perennial lpk folder

BRAT completed but layer packages need clipped: North Fork John Day 17070202 Lower John Day 17070204 North Fork Burnt 17050202

The following huc needed to be rerun due to discrepancies in the BRAT construction and Middle Fork John Day 17070203

Mic and I will be clipping layer packages today. Chalese is working on the summary products.

Albonicomt commented 6 years ago

@MatthewMeier, @wally-mac, and @joewheaton. The Perennial Lpks for the John Day will be housed here on box: https://usu.box.com/s/0jro2ebyz9of2tvhl097728yg6f717rf

MatthewMeier commented 6 years ago

@joewheaton @wally-mac @Albonicomt @CHafen Hello all, I have completed all but one of the analysis for the Project and am in the process of uploading the data to box now. I will provide links to where they will reside after I do a bit of organizing. Keep you all posted

MatthewMeier commented 6 years ago

@joewheaton @wally-mac I Have uploaded and reorganized the data in hopes that it will be more navigable for you all. Please provide feedback if you would prefer it to be different. I tried looking at the organization that the lab has for folder structure but I probably missed something. The following are links to all of the data and Perennial LPK that will be used for map making by @Albonicomt and @CHafen. I have also ran the preliminary analysis in excel spreadsheets that are needed for figure making. I hope with the spreadsheets that it will minimize the work to be done for maps. I will include links to all of these. Also let me know if you would like me to take any maps on. Thank you all.

John Day Perennial LPK EXCEL @Albonicomt can you double check my calculations on Upper John Day Excel sheet (I included my formulas in that one)

Upper John Day 17070201 North Fork John Day 17070202 Middle Fork John Day 17070203 Lower John Day 17070204

North Fork Burnt Perennial LPK EXCEL

North Fork Burnt 17050202 North Fork Burnt with Roads clipped down to major ones 17050202 Sorry about the delay it will not happen again. I will be available tomorrow and early next week for changes or help pushing the product out.

MatthewMeier commented 6 years ago

North burnt with roads clip had an error in the run fixing it now

MatthewMeier commented 6 years ago

It is all done and uploaded

MatthewMeier commented 6 years ago

@CHafen and @Albonicomt @wally-mac and I have looked over the data this morning and for all of the existing and historic capacity, and management outputs we are feeling really good about. We are going to explore doing a subset of roads later on, but as of right now full steam ahead making the maps for the John day and the North Fork Burnt.

@Albonicomt I believe you are doing the North Fork Burnt watershed maps and you will notice there is a Roads clipped BRAT run and output for this as watershed. What we did here was clip down the roads layer to capture only the major hwy and interstates that we could see. We did this because there are many old logging and forest service roads which are captured in the roads layer for the initial run of the North Fork Burnt. With all of these roads it can seem like you are having very high levels of human conflict everywhere you work. While there didn't seem to be a large difference in the outputs from looking at the stats which would warrant making BRAT output maps for both. We do feel it would be useful to put up a map showing the distance to closest infrastructure or distance to road under the human beaver conflict group/folder. Call if you have any questions and follow the links in the comment above for where to find the data.

Albonicomt commented 6 years ago

@MatthewMeier Sounds good! I am on it!

Albonicomt commented 6 years ago

@CHafen, @wally-mac, @joewheaton, @MatthewMeier Do we like this as a template for the Summary Products for BRAT. In the Chart, Keep the Kilometers Bar on x-axis, or get rid of it? Does the Chart along with @CHafen 's Legend display the information clearly? Any thoughts? I will roll with what I have if all seems good.
nf_burnt_river_brat_existing_dam_complex_size

Albonicomt commented 6 years ago

Whoops! Km definitely doesn't match up. My bad. I will get rid of that.

Albonicomt commented 6 years ago

nf_burnt_river_brat_existing_dam_complex_size

wally-mac commented 6 years ago

Looking good. Here are my suggested edits... I think that you should add as a second line under North Fork Burnt River 'Huc 10 with the number'. I think that it should add 'Perennial' before Stream Length. I find it to be a bit packed in there. I would say reduce the size of the watershed map a bit so the title and the bar chart box are not so close to the HUC 8 boundary. If this is unclear I can make a video of it. Please share an update version once you have made these changes.

On Mon, Oct 15, 2018 at 11:41 AM Micael Albonico notifications@github.com wrote:

[image: nf_burnt_river_brat_existing_dam_complex_size] https://user-images.githubusercontent.com/39168968/46967926-355c7880-d06f-11e8-84b6-c96dc76bd21f.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-429946764, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUjWAu-Ro1NSQIks7mR99p4SwbjSgks5ulMjAgaJpZM4XED_T .

-- Wally Macfarlane 435.512.1839

CHafen commented 6 years ago

It's hard to get the Km and miles on there and have them match up. As per Konrad's suggestion here is an example chart I made (just from the Logan maps that I already have). example_chart I just rounded to the nearest mile and added those to the bottom and put the Km on top, smaller an italic. For the actual charts I would use nice round numbers for the miles and round the Km to the nearest whole number I think this is the simplest way to do things while having it look nice. I think it is also helpful for doing it this way to not just have the tick marks at the bottom of the map, but a line all the way through as the example I'm sharing has. @wally-mac @joewheaton what are your thoughts?

Albonicomt commented 6 years ago

@wally-mac, Makes sense. I'll get on it.

@CHafen I really like the clean look of that Chart. Looks nice and simple. My vote is on your style.

wally-mac commented 6 years ago

Yes, @Chalese Hafen chalese.hafen@gmail.com I think that works well. Way-to-go!

On Mon, Oct 15, 2018 at 11:58 AM Micael Albonico notifications@github.com wrote:

@wally-mac https://github.com/wally-mac, Makes sense. I'll get on it.

@CHafen https://github.com/CHafen I really like the clean look of that Chart. Looks nice and simple. My vote is on your style.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-429952784, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUo2wQRkIDDqHQVty-CDTxXqs_tBtks5ulMysgaJpZM4XED_T .

-- Wally Macfarlane 435.512.1839

CHafen commented 6 years ago

@wally-mac on both the management outputs one of the categories is NA. In the layer package I got from Matt the outputs aren't showing the sections that are categorized as NA. I'm guessing we do want to show these portions of the stream on the map, right? If so, do you have suggestions for how it should look and what we should call them in the legends?

wally-mac commented 6 years ago

Hi Chalese, good question. I'm going to defer to @MatthewMeier who has been working with these data a ton in the past few days. Matt please see above.

Albonicomt commented 6 years ago

@wally-mac , updated Summary nf_burnt_river_brat_existing_dam_complex

CHafen commented 6 years ago

Hey Mic, I'd use the miles to make the chart and round to the nearest Km so those are the ones that are the weird numbers. The one I made is backwards because I just used what I already had. Also make the Km italic.

CHafen commented 6 years ago

Also, will the scale bar fit if you move it all the way to the right? You could try putting the N arrow above the scale bar if that fits better.

Albonicomt commented 6 years ago

@CHafen, right on! Thanks Chalese! Will do!

wally-mac commented 6 years ago

Here https://youtu.be/bc2-zlYB1FI is a short video that shows my suggested edits. Give these edits a try and post a new map. Thanks

On Mon, Oct 15, 2018 at 1:14 PM Micael Albonico notifications@github.com wrote:

@CHafen https://github.com/CHafen, right on! Thanks Chalese! Will do!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-429978864, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUk3PH1hNQOsBmwmflxbS1I8TZ010ks5ulN6dgaJpZM4XED_T .

-- Wally Macfarlane 435.512.1839

Albonicomt commented 6 years ago

Thanks for the Video @wally-mac and for the suggestions @CHafen. Here is an example of the Map. I'll have to run the Huc 10 through some excel calculations, so Minus the Analysis on the Huc-10 NF Burnt, this is what it will look like. nf_burnt_river_brat_existing_dam_complex

wally-mac commented 6 years ago

Nice work! I think the bar chart area is a bit large and it makes the scale bar look a bit cramped. Is there anything you can do about it?

On Mon, Oct 15, 2018 at 1:49 PM Micael Albonico notifications@github.com wrote:

Thanks for the Video @wally-mac https://github.com/wally-mac and for the suggestions @CHafen https://github.com/CHafen. Here is an example of the Map. I'll have to run the Huc 10 through some excel calculations, so Minus the Analysis on the Huc-10 NF Burnt, this is what it will look like. [image: nf_burnt_river_brat_existing_dam_complex] https://user-images.githubusercontent.com/39168968/46974308-e7507080-d080-11e8-936e-f3e0bcc61024.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/184#issuecomment-429989192, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUi1LRthuUjrvMbZfdWbnpDvCZHiqks5ulObNgaJpZM4XED_T .

-- Wally Macfarlane 435.512.1839

Albonicomt commented 6 years ago

I can reduce the size a bit, like below. I can reduce further, but I am not sure how much smaller we would want to go.

nf_burnt_river_brat_existing_dam_complex

Riverscapes / pyBRAT

BRAT Tasks: BRAT model runs due 10/8/18 and summary products due 10/15/18 #184