pyBRAT project curation.

Cashe93 commented 3 years ago

Description

For legacy pyBRAT runs:

Update the XSD to the correct url

<Project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://xml.riverscapes.xyz/Projects/XSD/V1/pyBRAT.xsd">

Change from <BRAT> to <pyBRAT> and productVersion should start lowercase

<!-- FROM -->
<BRAT ProductVersion="3.0.21" dateCreated="2019-05-19T16:27:20.860000" guid="1A274B27-B1B4-4782-A343-0B7C9251EF40" id="RZ1">

  <!-- TO -->
<pyBRAT productVersion="3.0.21" dateCreated="2019-05-19T16:27:20.860000" guid="1A274B27-B1B4-4782-A343-0B7C9251EF40" id="RZ1">

<Inputs>

Move everything from the top inputs down into the main one.
Remove anything with a ref="IDNAME"
Flatten <DrainageNetworks and use the <Vector> tag

       <Vector guid="7FDD19E5-1301-4F8C-A342-523B9BFA75D6">
         <Name>30m Buffer</Name>
         <Path>Outputs\Output_01\01_Intermediates\01_Buffers\buffer_30m.shp\</Path>
       </Vector>
       <Vector guid="97C58752-80EF-42FF-9D31-D1BD1C706034">
         <Name>100m Buffer</Name>
         <Path>Outputs\Output_01\01_Intermediates\01_Buffers\buffer_100m.shp\</Path>
       </Vector>

Flatten <Intermediates>

<!--- FROM --->
<Intermediates>
  <Intermediate>
    <Name>BRAT Intermediate</Name>
    <Vector guid="9DC9AA99-1C6E-4CD8-A33B-B6D6F5874F6C">
      <Name>BRAT Input Table</Name>
      <Path>Outputs\Output_01\01_Intermediates\BRAT_Table.shp\</Path>
    </Vector>
  </Intermediate>
</Intermediates>

<!--- TO --->
<Intermediates>
  <Vector guid="9DC9AA99-1C6E-4CD8-A33B-B6D6F5874F6C">
      <Name>BRAT Input Table</Name>
      <Path>Outputs\Output_01\01_Intermediates\BRAT_Table.shp\</Path>
    </Vector>
</Intermediates>

<Analyses />
1. Combine all the <Analyses> tags into one
2. Wrap <Vector> tags in <Outputs />
Misc Cleanup:
1. replace all file slashes: \ with /
2. Remove any slashes at the end of file paths
Add names and MetaData

Change the <Name> tag andd anything else that will help us identify this in the system. Metadata to the project, especially if there going to be more than one batch to upload. Give us some information about

<?xml version="1.0" ?>
<Project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://xml.riverscapes.xyz/Projects/XSD/V1/pyBRAT.xsd">

  <!-- <Name>Idaho_BRAT</Name>-->
  <Name>Idaho_BRAT HUC 17040206 Batch 01</Name>

  <ProjectType>BRAT</ProjectType>
    <MetaData>
        <Meta name="HUCID">17040206</Meta>
        <Meta name="Watershed">AmericanFalls</Meta>

    <!-- ADD YOUR EXTRA METADATA IF NEEDED -->
    <Meta name="Batch">01</Meta>
    <Meta name="BatchDescriptin">This batch describes the running of the tool for ...... blah blah blah</Meta>

  </MetaData>
  ...

Projects

These are the legacy projects that have a project.xml associated with them

[x] Idaho BRAT (84 projects)
[x] TNC BRAT (81 projects)
[ ] Montana BRAT (1 project)
[ ] GYE
[ ] John Day
[ ] Powder Basin
[ ] Panther Creek
[x] Sprague

To-Do

[x] Get pyBRAT Business Logic up to date
[x] #209
[x] #210
[x] #211
[x] Get a pyBRAT project to display correctly in RAVE
[ ] Finish editing project.xmls for all legacy pyBRAT runs

MattReimer commented 3 years ago

Hey @Cashe93 ,

I found a step that got omitted. You might want to add it to the instructions.

    <ProjectType>BRAT</ProjectType>

<!-- should become ... -->

    <ProjectType>pyBRAT</ProjectType>

It's an easy change but I'm still sorry I forgot to include it with the original instructions

wally-mac commented 3 years ago

@Cashe93 I just wanted to make sure you are pulling the data from where the final data lives. IDAHO: Idaho_PyBRAT CALIFORNIA TNC: California_PyBRAT

wally-mac commented 3 years ago

@Cashe93 here are the locations on Box of additional legacy projects that have a project.xml associated with them that will want run.
GYE: GYE_PyBRAT JOHN DAY: John_Day_PyBRAT POWDER BASIN: Powder_Basin_BRAT PANTHER CREEK: Panther_Creek_PyBRAT

wally-mac commented 3 years ago

@Cashe93 as we talked about in our meeting today let's prioritize running PyBRAT for the SPRAGUE: Sprague_PyBRAT

Cashe93 commented 3 years ago

@joewheaton, the pyBRAT business logic has been updated and is functioning. Aside from a couple of problematic layers (almost exclusively being categorized rasters) things look good. You can view the RAVE layers in #209, #210 , #211. All changes including changes to QMLs have been committed to #207.

joewheaton commented 3 years ago

This is really amazing @Cashe93 to have all these looking so good. Nice work! So, does this mean if you open up any of the pyBRAT projects they seem to be working other than a few layers? Or does it mean that you open up any of the pyBRAT projects with some manual edits to the project.rs.xml file and it works?

Either way, this is a big deal. Great Job.

Cashe93 commented 3 years ago

@joewheaton, the latter is correct. Unfortunately, most of the legacy pyBRAT project XMLs still need to be manually edited/validated before they will display properly in RAVE. This is something that I am currently working on and that Ally will be assisting me with once she gets oriented with the lab. But the functioning business logic will make this task a bit more straightforward.

Cashe93 commented 3 years ago

@wally-mac, just a heads up, the IDF&G and GYCC projects that are located within the NonProject > etal_public_html folder don't have the project XMLs associated with them. However, they do exist within the Project > USA > Idaho folder. These are the ones I've been validating/editing.

Cashe93 commented 3 years ago

@philipbaileynar or @MattReimer, Since @leallysmith can't yet view the pyBRAT projects in RAVE, I've been going in and making sure that the edits she makes to the project XMLs have been working. For the most part, everything is looking great, however, I've been running into a weird issue. See short video here which briefly explains the issue and this related video that recreates the issue from start to finish.

Cashe93 commented 3 years ago

@Cashe93 as we talked about in our meeting today let's prioritize running PyBRAT for the SPRAGUE:

@wally-mac the pyBRAT run for the Sprague is good to go. You can find it here on box.

Cashe93 commented 2 years ago

@MattReimer, as discussed before, thanks to the work of @leallysmith, the legacy project XMLs have been edited for the majority of these and should be good to go. Let me know when you can meet to discuss further steps.

Cashe93 commented 2 years ago

@MattReimer here are public links to three representative pyBRAT projects.

Cashe93 commented 2 years ago

@joewheaton, could you advise me on which of these projects need their own program, and which ones can be placed into existing programs?

Idaho BRAT (84 projects)
TNC BRAT (81 projects)
Montana BRAT (1 project)
GYE
John Day
Powder Basin
Panther Creek
Sprague

joewheaton commented 2 years ago

@joewheaton, could you advise me on which of these projects need their own program, and which ones can be placed into existing programs?

Sure. These are all for pyBRAT projects that we'll upload I am assuming.

Project	HUC 8 Project Count	Exists or Need New	Program Name (link if existing)	Program URL (link if existing)
Idaho BRAT	84	New	Idaho Department of Fish & Game	IDFG
TNC BRAT		New	TNC California	TNC_CA
Monanta BRAT	1	Exists	BLM Montana and Dakotas	BLM
GYE BRAT	20	New	Greater Yellowstone Coordinating Committee	GYE
John Day BRAT	5 ?	New	John Day Watershed Council	JDWC
NRCS BRAT Runs	15-20 ?	New	SGI Mesic Workshops	SGI
Powder Basin BRAT	2	New	Powder Basin Watershed Council	PBWC
Panther Creek BRAT	1	Exists	Anabranch	Anabranch
Sprague BRAT	1-3?	Exists	Upper Klamath - USFWS	UKUSFWS

Cashe93 commented 2 years ago

Awesome! Thanks, @joewheaton

Cashe93 commented 2 years ago

@joewheaton and @MattReimer

Update: IDFG, TNC_CA, GYE, JDWC, PBWC and UKUSFWS have been uploaded. However the following could not be uploaded for various reasons.

Panther Creek: The folders and layers exist, but there is currently no project.xml.
NRCS: We were not aware of these runs until recently, so we just need a bit more time to complete the edits.
Montana BRAT: the folders are there, but the actual layers that the project.xml is looking for don't exist. This is quite odd, as the companion files such as cpg, prj, sbn, sbx ect. exist, but the .shp and the .tif files are missing for most of the outputs and intermediates. I've also explored the folder labeled 'OLD' but it appears to be the same.

wally-mac commented 2 years ago

@Cashe93 can you please update what we know about the Montana run and close the ticket. Panther Creek and NRCS can be taken care of later.

joewheaton commented 2 years ago

@wally-mac this should not be closed until these are done. Does @MattReimer know to produce programs for all of these? As per https://github.com/Riverscapes/RiverscapesData/issues/104 Q7 and Q8 the only ones we really care about for today's fire drill are potentially Idaho BRAT, GYA BRAT and Montana BRAT. The others can wait, but don't close this ticket unless you guys make seperate tickets for those.

Thanks.

Cashe93 commented 2 years ago

@joewheaton in response to Q7 and Q8 in Riverscapes Data issue #104, GYE and IDFG pyBRAT projects have been edited and uploaded to the S3 server, but still need to be uploaded to the Warehouse. @MattReimer, is this correct?

MattReimer commented 2 years ago

100% correct. I am the bottleneck here. Sorry about that.

MattReimer commented 2 years ago

Regarding:

https://github.com/Riverscapes/RiverscapesData/issues/87

This is done.

Riverscapes / RiverscapesXML