Add to pyBRAT BL the missing "Vegetation Suitability" intermediates to match sqlBRAT

[Cashe93]

@joewheaton, I'm having difficulty adding this layer to the pyBRAT intermediates. It appears to be an issue with how the BL is referencing the project XML. Here is a short video going over the issue and a potential solution. Let me know your thoughts. The video also points out that since pyBRAT is just applying the suitability symbology to the existing vegetation raster, we can anticipate having the same categorical raster issue as mentioned in #393.

[Cashe93]

@MattReimer, is there a workaround here? I'm trying to add a layer into the Intermediates node within the BL; the node begins with this line:

 <Node label="Intermediates" xpath="Realizations/pyBRAT/Intermediates">

But since the layer I'm trying to add is within the project XMLs input node ( xpath="Realizations/pyBRAT/Inputs"), it can't seem to find what I'm trying to referene. Does this make sense?

[MattReimer]

Can you paste the whole file in please. Much easier if I can see it in relation to everything else.

[Cashe93]

Here is the BRAT BL:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://xml.riverscapes.xyz/RaveBusinessLogic/XSD/project_explorer.xsd">
  <Name>pyBRAT</Name>
  <ProjectType>pyBRAT</ProjectType>
  <Node xpathlabel="Name">
    <Children collapsed="false">
      <Node label="Outputs" xpath="Realizations/pyBRAT/Analyses/Analysis/Outputs">
        <Children collapsed="true">
          <Node xpathlabel="Name" xpath="SQLiteDB" type="file" />
          <Node xpathlabel="Name" xpath="HTMLFile" type="report" />
          <Node label="Capacity">
            <Children>
              <Node label="Existing Dam Capacity" xpath="Vector[@id='COMBINED_CAPACITY']" type="line" symbology="oCC_EX" id="ex_capacity" />
              <Node label="Existing Dam Size" xpath="Vector[@id='COMBINED_CAPACITY']" type="line" symbology="mCC_EX_CT" />
              <Node label="Historic Dam Capacity" xpath="Vector[@id='COMBINED_CAPACITY']" type="line" symbology="oCC_HPE" id ="his_capacity"/>
              <Node label="Historic Dam Size" xpath="Vector[@id='COMBINED_CAPACITY']" type="line" symbology="mCC_HPE_CT" />
            </Children>
          </Node>
          <Node label="Management">
            <Children collapsed="true">
              <Node label="Unsuitable or Limited Opportunities" xpath="Vector[@id='OPPORTUNITY']" type="line" symbology="oPBRC_UD" id="limited" />
              <Node label="Risk of Undesirable Dams" xpath="Vector[@id='RISK']" type="line" symbology="oPBRC_UI" id="risk" />
              <Node label="Restoration or Conservation Opportunities" xpath="Vector[@id='OPPORTUNITY']" type="line" symbology="oPBRC_CR" id="opportunity" />
            </Children>
          </Node>
          <Node label="Validation">
            <Children collapsed="true">
              <Node label="Predicted Dam Density vs. Surveyed Dam Density" xpath="Vector[@id='DATA_VALIDATION']" type="line" symbology="predicted_surveyed" id=""/>
              <Node label="Percent of Historic Dam Capacity Remaining" xpath="Vector[@id='DATA_VALIDATION']" type="line" symbology="percent_remaining" id=""/>
              <Node label="Current Beaver Dam Management Strategies" xpath="Vector[@id='DATA_VALIDATION']" type="line" symbology="current_strategies" id=""/>
            </Children>
          </Node>
        </Children>
      </Node>
      <Node label="Intermediates" xpath="Realizations/pyBRAT/Intermediates">
        <Children collapsed="true">
          <Node label="Perennial">
            <Children>
              <Node label="Perennial" xpath="Vector[@id='FULL_INT']" type="line" symbology="IsPeren" />
            </Children>
          </Node>
          <Node label="Topographic Index">
            <Children>
              <Node label="Slope" xpath="Vector[@id='FULL_INT']" type="line" symbology="iGeo_Slope" id="slope" />
              <Node label="Drainage Area" xpath="Vector[@id='FULL_INT']" type="line" symbology="iGeo_DA" id="DA" />
            </Children>
          </Node>
          <Node label="Anabranch Handler">
            <Children>
              <Node label="Anabranch Types" xpath="Vector[@id='FULL_INT']" type="line" symbology="IsMainCH" />
            </Children>
          </Node>
          <Node label="Hydrology">
            <Children>
              <Node label="High Flow Stream Power" xpath="Vector[@id='FULL_INT']" type="line" symbology="iHyd_SP2" id="flow_high" />
              <Node label="Base Flow Stream Power" xpath="Vector[@id='FULL_INT']" type="line" symbology="iHyd_SPLow" id="flow_base" />
            </Children>
          </Node>
          <Node label="Overall Vegetation Dam Capacity">
            <Children>
              <Node label="Existing Veg Dam Building Capacity" xpath="Vector[@id='FULL_INT']" type="line" symbology="oVC_EX" />
              <Node label="Historic Veg Dam Building Capacity" xpath="Vector[@id='FULL_INT']" type="line" symbology="oVC_HPE" />
            </Children>
          </Node>
          <Node label="Anthropogenic Intermediates">
            <Children>
              <!-- <Node label="Priority Beaver Translocation Areas" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_Privat" /> -->
              <Node label="Land Ownership per Reach" xpath="Vector[@id='FULL_INT']" type="line" symbology="PY_ADMIN" />
              <Node label="Land Use Intensity" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_LU" />
              <Node label="Distance to Nearest Infrastructure" xpath="Vector[@id='FULL_INT']" type="line" symbology="oPC_Dist" />
              <Node label="Distance to Railroad" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_Rail" />
              <Node label="Distance to Railroad in Valley Bottom" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_RailVB" />
              <Node label="Distance to Road" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_Road" />
              <Node label="Distance to Road Crossing" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_RoadX" />
              <Node label="Distance to Road in Valley Bottom" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_RoadVB" />
              <Node label="Distance to Points of Diversion" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_DivPts" />
              <Node label="Distance to Canals/Ditches" xpath="Vector[@id='FULL_INT']" type="line" symbology="iPC_Canal" />
            </Children>
          </Node>
          <Node label="Buffers">
            <Children>
              <Node label="30m Buffer" xpath="Vector[@id='BUF_30']" type="polygon" symbology="buf30" />
              <Node label="100m Buffer" xpath="Vector[@id='BUF_100']" type="polygon" symbology="buf100" />
            </Children>
          </Node>
        </Children>
      </Node>
      <Node label="Inputs" xpath="Realizations/pyBRAT/Inputs">
        <Children collapsed="true">
          <Node label="Network">
            <Children>
              <Node label="Network" xpath="Vector[@id='FLOWLINES']" type="line" symbology="py_flow_lines" />
              <Node label="Flow Direction" xpath="Vector[@id='FLOWLINES']" type="line" symbology="py_FlowDirection" />
            </Children>
          </Node>
          <Node label="Anthropogenic Layers">
            <Children>
              <Node label="Provisional Points of Diversion" xpath="Vector[@id='DIVERSION_PTS']" type="point" symbology="py_div_point" />
              <Node label="Canals and Ditches" xpath="Vector[@id='CANAL']" type="line" symbology="canals_ditches" />
              <Node label="Railroads" xpath="Vector[@id='RR']" type="line" symbology="railroads" />
              <Node label="Roads" xpath="Vector[@id='ROAD']" type="line" symbology="py_roads" />
              <Node label="Land Ownership" xpath="Vector[@id='OWNERSHIP']" type="polygon" symbology="py_ownership" transparency="40" />
              <Node label="Valley Bottom Extent (Hollow)" xpath="Vector[@id='VALLEY']" type="polygon" symbology="vbet68_hollow" />
              <Node label="Valley Bottom Extent (Filled)" xpath="Vector[@id='VALLEY']" type="polygon" symbology="vbet68_filled" />
              <Node label="Land Use" xpath="Raster[@id='LU']" type="raster" symbology="py_lu" transparency="40"/>
            </Children>
          </Node>
          <Node label="Vegetation">
            <Children>
              <Node label="Existing Vegetation" id="vegetation">
                <Children>
                  <Node label="Veg Type - EVT Class" xpath="Raster[@id='EXVEG']" type="raster" symbology="Existing_Veg_EVT_Class" transparency="40" />
                  <Node label="Veg Type - EVT Name" xpath="Raster[@id='EXVEG']" type="raster" symbology="Existing_Veg_EVT_Name" transparency="40" />
                  <Node label="Existing Riparian" xpath="Raster[@id='EXVEG']" type="raster" symbology="Existing_Veg_EVT_Riparian" transparency="40" />
                </Children>
              </Node>
              <Node label="Historic Vegetation">
                <Children>
                  <Node label="Veg Type - BPS Name" xpath="Raster[@id='HISTVEG']" type="raster" symbology="Historic_Veg_BPS_Name" transparency="40" />
                  <Node label="Historic Riparian" xpath="Raster[@id='HISTVEG']" type="raster" symbology="Historic_Veg_BPS_Riparian" transparency="40" />
                </Children>
              </Node>
            </Children>
          </Node>
          <Node label="Topography">
            <Children>
              <Node label="Flow Accumulation" xpath="Raster[@id='DR']" type="raster" symbology="py_flow_accumulation" transparency="40" />
              <Node label="Slope" xpath="Raster[@id='SLOPE']" type="raster" symbology="slope" transparency="40" />
              <Node label="DEM" xpath="Raster[@id='DEM']" type="raster" symbology="dem" transparency="40" />
              <Node label="Hillshade" xpath="Raster[@id='HS']" type="raster" symbology="hillshade" id="hillshade" />
            </Children>
          </Node>
        </Children>
      </Node>
    </Children>
  </Node>
  <Views default="BRAT_exOutputs">
    <View name="BRAT Existing Capacity" id="BRAT_exOutputs">
      <Description>This view is the existing dam capacity</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="ex_capacity" />
      </Layers>
    </View>
    <View name="BRAT Historic Capacity" id="BRAT_hisOutputs">
      <Description>This view is the historic dam capacity</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="his_capacity" />
      </Layers>
    </View>
    <View name="BRAT Limited Opportunities" id="BRAT_limited">
      <Description>This view shows unsuitable or limited opportunities</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="limited" />
      </Layers>
    </View>
    <View name="BRAT Risk of Undesireable Dams" id="BRAT_risk">
      <Description>This view shows risk of undesireable dams</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="risk" />
      </Layers>
    </View>
    <View name="BRAT Restoration or Conservation" id="BRAT_opportunity">
      <Description>This view shows restoration/conservation opportunities</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="opportunity" />
      </Layers>
    </View>
    <View name="BRAT Slope" id="BRAT_slope">
      <Description>This view shows network slope</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="slope" />
      </Layers>
    </View>
    <View name="BRAT Drainage Area" id="BRAT_DA">
      <Description>This view shows drainage area</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="DA" />
      </Layers>
    </View>
    <View name="BRAT High Flow" id="BRAT_highflow">
      <Description>This view shows high flow stream power</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="flow_high" />
      </Layers>
    </View>
    <View name="BRAT Base Flow" id="BRAT_baseflow">
      <Description>This view shows base flow stream power</Description>
      <Layers>
        <Layer id="hillshade" />
        <Layer id="flow_base" />
      </Layers>
    </View>
  </Views>
</Project>

In short, I'm trying to add a layer with xpath="Raster[@id='EXVEG'] into the intermediates node.

[MattReimer]

Ok, I see. If you really need to add something from the inputs folder into the intermediates section the only option is to reduce the context of the top node like so:

      <Node label="Intermediates" xpath="Realizations/pyBRAT/Intermediates">
     To:
      <Node label="Intermediates" xpath="Realizations/pyBRAT/">

And then add /intermediates to every sublayer expect the one that comes from inputs.

Does that make sense?

[Cashe93]

Yes, that makes sense. Thanks, @MattReimer.

[Cashe93]

@MattReimer I have a question that applies to both this layer and the Land Use raster that we discussed (and I suppose all categorized rasters). For the Veg Suitability webRAVE symbology, I have this txt file:

# QGIS Generated Color Map Export File
INTERPOLATION:EXACT
4,0,97,0,102,Preferred
3,122,171,0,102,Suitable
2,255,255,0,102,Moderately Suitable
1,255,153,0,102,Barely Suitable
0,255,34,0,102,Unsuitable

However, the values (0-4) are found in the attribute field 'VEG_CODE'. But there is no way to reference that attribute in Q or Web, correct? So the WebRAVE symbologies that I created for Land Use and Veg Suitability likely won't be applied to the rasters? We can set up a meeting to discuss this further if needed, but as of now, I can only successfully curate symbology for categorized rasters in ArcRAVE.

[MattReimer]

Oh, I see. You're not actually referencing the raster values but another attribute in the RAT.

You have two choices here:

1. Manually create a table with every possible raster value in the value column and the desired label on the left. Take a look at Existing_Veg_EVT_Class.txt for an example of how that works.
2. You'll recall that I created some custom python code in a jupyter notebook to read the RAT table for Landfire and then generate the necessary.clr lookup files for webRAVE and QGis

In order to use option 2 you're going to need a raster with a complete RAT that has every possible value inside it. Otherwise it won't work.

Let me know what you choose. If you choose to go automated then maybe we can pull @jtgilbert in to help. The python code is pretty simple and shouldn't take much work to extend.

[Cashe93]

@MattReimer, I would like to go the automated route if possible. Good opportunity to get some hands-on with this stuff.

[Cashe93]

@jtgilbert, when you have time, would you be willing to go over the code and walk through this process with me?

[jtgilbert]

@Cashe93 Yep, I'll try and get to this within the next couple days and reach out when ready.

[jtgilbert]

@Cashe93 I exteded the jupyter notebook @MattReimer mentioned, and generated a .clr file for coding land use intensity based on the evt raster. I also used it to create a .qml layer file. The changes are on this branch: https://github.com/Riverscapes/RiverscapesXML/tree/lui-symbology

If you want, we could video chat and I could explain what I did. What still needs done is:

• [x] create the .txt file for webrave symbology
• [x] create the .lyr file for arcrave symbology

We should also link #418 here

[Cashe93]

Thanks, @jtgilbert. I don't have time today, but perhaps later in the week we can go over what you did. I can create the .txt and .lyr in the meantime. Note: this fix will only be applied to pyBRAT projects.

[Cashe93]

@jtgilbert, have you done this for Vegetation Suitability as well? Or just Land Use for now? I'll go ahead and create a pull request for the Land Use symbology changes.

[jtgilbert]

@Cashe93 I'll work on the veg suitability ones this afternoon hopefully.

[jtgilbert]

Hey @Cashe93 the new branch is here: https://github.com/Riverscapes/RiverscapesXML/tree/lui-suitability-symbology

I took it to the same place as the last one (i.e., created qml files), but this time there's one for land use and one for veg suitability.

[Cashe93]

Thanks, @jtgilbert! I'll get .lyr and .txt files for these produced tomorrow.

[Cashe93]

I've noticed a potential issue here; but it might not be worth the trouble. It looks like the raster 'value' attribute field isn't consistent for all projects. I think it might be caused from using older versions of LANDFIRE, but I'm not sure. See this short video.

[jtgilbert]

Hey @Cashe93, good catch, the way I generated the land use intensity was from the most current landfire that has those values >7000. I think I generated the veg suitability from a table that had both old and new landfire values. I don't know if it's worth worrying about....I'm not sure what the overall game plan is in terms of supporting older landfire layers.

[Cashe93]

@jtgilbert and @MattReimer, the Web symbology does not appear to be working for Land Use or Veg Suitability. Its working for both Q and Arc though.

Project: https://webrave.riverscapes.xyz/#/IDFG/2e66e375-891e-420e-907e-7cc2a67a812c

Land Use

Existing Veg Suitability

Historic Veg Suitability

[jtgilbert]

@Cashe93 Weird, I can't figure it out just from a quick look. I did notice though that existing veg suitability looks different in Q and Arc. Are you sure that the lyr file matches the qml? Maybe I generated the qml off of outdated veg suitability scores...

[Cashe93]

@jtgilbert the symbology is generally the same, but since ArcMap doesn't have an issue with RATs, the .lyr is set up much more simply than the .qml.

Compared to:

The only difference I am seeing is that in Arc, open water is coded as 0 or unsuitable, and in Q it's being picked up as preferred. Is this the difference you are seeing, or are you seeing more discrepancies? I believe coding open water as 4 (preferred) is what we want, so the .qml you produced seems to be correct. In the past, however, they coded open water as 0 (unsuitable), which is why in ArcMap, open water is showing up as unsuitable, at least for this project; because the VEG_CODE attribute for open water = 0. Does that make sense? It's a bit of a mess, and I'm not sure how to go about fixing it.

[jtgilbert]

@Cashe93 I was noticing conifer classes that have a VEG_CODE of 3 in the landfire attribute table, but a default suitability of 2 in the databases and other tables in the repository.

[Cashe93]

@jtgilbert, I think the default suitability scores may have been updated a bit since these pyBRAT projects have been run. If you are comparing pyBRAT suitabilities to the default suitabilities we are currently using for sqlBRAT, that might be the cause for the differences you are seeing.