Project Structure Revision

[banderson1618]

Mic and I have been been assigned with figuring out what the folder structure of pyBRAT should look like, including showing intermediary data.

My rough draft for folder structure is laid out below, with bullet points.

• Inputs
  • Conflict Layers
  • Valley Bottom
  • Roads
  • Railroads
  • Canals
  • Land Use Raster
  • Network
  • Stream Network
  • Vegetation
  • Existing Vegetation
  • Historic Vegetation
  • Topography
  • DEM
  • Slope
  • Flow Accumulation
  • Hillshade
• Intermediates
  • Buffers
  • 30 m Buffer
  • 100 m Buffer
  • BRAT Table output
  • BRAT Braid Handler output
  • iHyd Streamflow Attributes output
  • BRAT Vegetation Dam Capacity Model output
• Output
  • BRAT Combined Dam Capacity Model output
  • BRAT Conflict Potential output
  • BRAT Conservation and Restoration Model output
  • Summary Report output

We could potentially put the Slope, Flow Accumulation, and Hillshade into the Intermediates folder, since we calculate them in the tool. Both seem reasonable to me.

I'll be having more conversations with Mic over email, since (to my knowledge) he does not have a GitHub account, but I'll do my best to keep this page updated with what we're working on, and I'll link my commits to it.

[Albonicomt]

I am here on Github. I like this general structure. I think it hits every step in the BRAT tool. Although, in my opinion, I think the slope, Flow accumulation, and hillshade work nicely in the Inputs under the Topography layer. I think it makes sense for the user to find those layers in the Topography layer. Thoughts for that?

Another thing we need to look at is creating a standard symbology for the BRAT table output, Ihyd outputs (Low Flow and high Flow), Veg Dam Cap, Braid Handler, Summary Report, and Land Use Raster.

[banderson1618]

For the intermediaries, it might make sense to not give them a symbology. Symbolizing based off of one attribute out of the many that are created implies that it's the most important one created, when I'm not sure that that's the case for, say, the BRAT Table output.

On the subject of slope, flow acc, and hillshade, in my mind, it makes sense to limit the Inputs folder to only be what the user gives to the tool. We could have a Topography folder in Intermediates, to make the association between the DEM and everything else more clear. I'm flexible on the issue, though.

@wally-mac, @MatthewMeier, and @bangen, do you have any comments on this issue?

[wally-mac]

I agree, limit the Inputs folder to only be what the user gives to the tool.

[Albonicomt]

Nice! I am all for simplicity. Any other thoughts on other standards for the symbologies in the Package?

[MatthewMeier]

Braden in this layer package at least holds the symbology for the Summary Report data "e_DamDens" it is basically the same symbology for the oCC_EX and oCC_PT layers

[banderson1618]

Another thought on the Topology intermediates is that we put them in both places. It would be space inefficient, but it makes sure that the user can find them, no matter where they think to look.

[MatthewMeier]

I think if you have the symbology in one location it makes it less confusing to the user because if it is in two places it is easy to think that duplicated symbologies are different from one another and which one to use becomes a question

[wally-mac]

Some additional intermediates included the segmented drainage network, the 30 meter network buffer, and the 100 meter network buffer. These are important because these make up the minimal mapping units of the tool and the associated outputs @banderson1618 @Albonicomt

[banderson1618]

The "segmented drainage network" is the same thing as the stream network, right? Would you prefer to put the stream network in the intermediates folder, or keep it in the inputs folder, @wally-mac?

[wally-mac]

I was thinking there was a segmented version and a unsegmented version but maybe we don't need the unsegmented version.

On Wed, May 16, 2018 at 10:00 AM, Braden Anderson notifications@github.com wrote:

The "segmented drainage network" is the same thing as the stream network, right? Would you prefer to put the stream network in the intermediates folder, or keep it in the inputs folder, @wally-mac https://github.com/wally-mac?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/45#issuecomment-389572839, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUlP85e8XOCqszb25lUVWb8_aX4-5ks5tzE0WgaJpZM4UATq6 .

-- Wally Macfarlane 435.512.1839

[MatthewMeier]

There is only a segmented drainage network. This is an input for the BRAT tool. Something to consider though with this is that now that we are running a complete network through the tool and then subsetting it later it might be beneficial to think about how we want to organize this for future projects. While the subset is all that I feel like we will be making in these layer packages this is something to think about.

[banderson1618]

Sounds good. I've updated my original post to have buffers in the intermediates folder

[banderson1618]

Something new to consider: how should "Output_#" work in this new system?

It seems to me that there are three possibilities:

We could put Output above Intermediate and Analyses folder, like below:

• Project Folder
  • Inputs
  • input folders
  • Output_1
  • Intermediates
    • Buffers
    • 30 m Buffer
    • 100 m Buffer
    • BRAT Table output
    • BRAT Braid Handler output
    • iHyd Streamflow Attributes output
    • BRAT Vegetation Dam Capacity Model output
  • Analyses
    • BRAT Combined Dam Capacity Model output
    • BRAT Conflict Potential output
    • BRAT Conservation and Restoration Model output
    • Summary Report output

We could put an Output folder for Intermediates and Analyses separately, like the following:

• Project Folder
  • Inputs
  • input folders
  • Intermediates
  • Output_1
    • Buffers
    • 30 m Buffer
    • 100 m Buffer
    • BRAT Table output
    • BRAT Braid Handler output
    • iHyd Streamflow Attributes output
    • BRAT Vegetation Dam Capacity Model output
  • Analyses
  • Output_1
    • BRAT Combined Dam Capacity Model output
    • BRAT Conflict Potential output
    • BRAT Conservation and Restoration Model output
    • Summary Report output

Finally, we could have an output folder for each subfolder, like the following:

• Project Folder
  • Inputs
  • input folders
  • Intermediates
  • Buffers
    • 30 m Buffer
    • Output_1
    • 100 m Buffer
    • Output_1
  • BRAT Table output
    • Output_1
  • BRAT Braid Handler output
    • Output_1
  • iHyd Streamflow Attributes output
    • Output_1
  • BRAT Vegetation Dam Capacity Model output
    • Output_1
  • Analyses
  • BRAT Combined Dam Capacity Model output
    • Output_1
  • BRAT Conflict Potential output
    • Output_1
  • BRAT Conservation and Restoration Model output
    • Output_1
  • Summary Report output
    • Output_1

Personally, I think the first solution makes the most sense. Because each step is tied to the next one, it seems important to me that they be grouped appropriately. The other two solutions have the potential for outputs to get "out of sync" with each other, in that Output 5 in Intermediates could potentially feed to Output 6. That many runs is unlikely, but it illustrates my point, I think.

Regardless, I want to get everyone's thoughts on this before I implement one of them. @wally-mac, @bangen, @MatthewMeier, @Albonicomt, and @joewheaton, do you have any feedback for what I should go ahead with?

[Albonicomt]

I like the first and third options the best. Although, I do see your point if we end up having several outputs.

[wally-mac]

I think the first solution is best.

On Wed, May 16, 2018 at 12:01 PM, Micael Albonico notifications@github.com wrote:

I like the first and third options the best. Although, I do see your point if we end up having several outputs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/45#issuecomment-389611486, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUl6eXyGw1hmcX5X2DhYo3BqpNQkPks5tzGmTgaJpZM4UATq6 .

-- Wally Macfarlane 435.512.1839

[banderson1618]

Unless someone speaks up, I'm going to go forward with implementing the first solution.

[wally-mac]

Go for it.

On Wed, May 16, 2018 at 1:06 PM, Braden Anderson notifications@github.com wrote:

Unless someone speaks up, I'm going to go forward with implementing the first solution.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/45#issuecomment-389631315, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUjSL37xMTAPCIn0IXqJuLXhcjgrUks5tzHidgaJpZM4UATq6 .

-- Wally Macfarlane 435.512.1839

[banderson1618]

Here is a picture of the tool's output, and the folder structure that is automatically created. The first one has some of the input folders collapsed, the other one has everything fully expanded.

Right now, the Braid Handler, iHyd, and Vegetation Dam Capacity all just add on attributes to the BRAT Table output, and so they don't show up as their own thing in the intermediate folder. We can copy them change that, for the sake of preserving that data, if that's our preference.

Let me know if there's anything else that we should change, @wally-mac, @Albonicomt, and @MatthewMeier.

There's some bug that's causing it to keep the Temp directory, but I'm working on a fix.

[wally-mac]

Please add @bangen to the conversation.

On Wed, May 16, 2018 at 4:27 PM, Braden Anderson notifications@github.com wrote:

Here is a picture of the tool's output, and the folder structure that is automatically created. The first one has some of the input folders collapsed, the other one has everything fully expanded.

Right now, the Braid Handler, iHyd, and Vegetation Dam Capacity all just add on attributes to the BRAT Table output, and so they don't show up as their own thing in the intermediate folder. We can copy them change that, for the sake of preserving that data, if that's our preference.

Let me know if there's anything else that we should change, @wally-mac https://github.com/wally-mac, @Albonicomt https://github.com/Albonicomt, and @MatthewMeier https://github.com/MatthewMeier.

[image: newstructuretestcollapsed] https://user-images.githubusercontent.com/21353496/40146969-e1ec7248-5924-11e8-85e3-c4420f659dbf.PNG

[image: newstructuretest] https://user-images.githubusercontent.com/21353496/40146874-7e065730-5924-11e8-96f0-5f7f34e147bf.PNG

There's some bug that's causing it to keep the Temp directory, but I'm working on a fix.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/45#issuecomment-389685596, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUgwXZ0CtYNDnRCWvrGTZxF6cFQ72ks5tzKfEgaJpZM4UATq6 .

-- Wally Macfarlane 435.512.1839

[Albonicomt]

You know another thing that would make creating these LPks much quicker (now that I think about it) would be to somehow automatically fill in a description into each layer. To create an LPK you have to fill in a description for every single Layer and group Layer you create. This process is pretty tedious, @banderson1618, do you think this would be easy to automate? We can talk about it later, but just something to note. Nice job on getting the Symbologies automated by the way!

[wally-mac]

Excellent idea! Please look into it.

On Thu, May 17, 2018 at 1:27 PM, Micael Albonico notifications@github.com wrote:

You know another thing that would make creating these LPks much quicker (now that I think about it) would be to somehow automatically fill in a description into each layer. To create an LPK you have to fill in a description for every single Layer and group Layer you create. This process is pretty tedious, @banderson1618 https://github.com/banderson1618, do you think this would be easy to automate? We can talk about it later, but just something to note. Nice job on getting the Symbologies automated by the way!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Riverscapes/pyBRAT/issues/45#issuecomment-389981302, or mute the thread https://github.com/notifications/unsubscribe-auth/AU-QUnUnaYViC2sMuBWbfYkyqd3YvbaMks5tzc80gaJpZM4UATq6 .

-- Wally Macfarlane 435.512.1839

[banderson1618]

@Albonicomt, I'll look into it. I remember trying to a while ago, and not having much luck, but I'll see what I can do.

[banderson1618]

@Albonicomt, I looked into it, and I made it work in testing. What should their description be? Something like "This is a layer file that is symbolized to show _____"?

[bangen]

Just making a quick note for anyone looking back at this thread that the 'BRAT Braid Handler output' wasn't added since it technically doesn't exist. The tool simply updates/overwrites the DA values in the BRAT network shapefile.

[banderson1618]

I've attached a picture of the folder structure that the tool currently produces. @wally-mac, @joewheaton, @MatthewMeier, @bangen, @Albonicomt, let me know what you think, and if we need to change anything. Otherwise, I'd like to close this issue.

[Albonicomt]

@banderson1618 Nice job man! That is sweet!

A few things I see offhand that would be quick to fix, is the new Conservation_Restoration_Model_V2 that Sara created. She has posted the symbology on Github, but I can send that to you if you need that Layer file.

Also, in the Vegetation folder (you might have these symbologies, but the folder isn't expanded in the screenshot, so if it is no worries) the VEG_CODE, Ex/Hist_Riparian symbologies.

Other than that, this looks sweet!