Anthropogenic layer and hydrologic calculation projects

[jtgilbert]

Create new project types for anthropogenic input layers and their derivatives, and for hydrologic (Q and SP) calculations. These projects would then be consumed by BRAT, instead of being part of BRAT, and could be consumed by other tools.

[philipbaileynar]

My $0.02 is that this just creates a data management headache as every step in our tools gets exploded into multiple projects that all need tracking and management. Nightmare. Nothing stopping those other tools from consuming the data from sqlBRAT projects.

[jtgilbert]

@philipbaileynar this was a suggestion from @joewheaton , your point here makes sense too, I'm sure he'll want to discuss.

[joewheaton]

My $0.02 is that this just creates a data management headache as every step in our tools gets exploded into multiple projects that all need tracking and management. Nightmare. Nothing stopping those other tools from consuming the data from sqlBRAT projects.

Yep, that's the overwhelmed @philipbaileynar speaking above.... Lets regroup and think more carefully about this now that we have a viable dataspec strategy that includes a much better plan for project references. I'd also value @MattReimer opinion on this one.

[MattReimer]

I expect the discussions around when a riverscapes dataset or group of datasets needs to become its own project type to be pretty much eternally fought. It's kind of part of the process and there's never going to be one answer for everyone.

We could very easily spend a lot of effort moving deck chairs around and arguing about logical groupings of layers. That's a totally valid thing to do but it has to balance with the "We already have it in this other project so just go get it there even if it's not totally ideal" conversation.

There is some effort required to create and maintain a new project type. When the value of the layers it would contain coupled with the discomfort of them not being in the "right" place surpasses the effort needed to create and maintain it then it might make sense to split it out.

Sorry if that's not a totally satisfying answer.

Practically, @philipbaileynar's right about a short-term headache. My immediate preference would be to not even tackle this until the warehouse refactor is done. After that we can revisit and certain things may be easier to do when our XML is a little more standard and our tools are a little more integrated.

[joewheaton]

Helpful insights @MattReimer - thanks. @jtgilbert FYI.
Re:

My immediate preference would be to not even tackle this until the warehouse refactor is done. After that we can revisit and certain things may be easier to do when our XML is a little more standard and our tools are a little more integrated.

I completely agree on waiting until refactor. We do not entirely know what project referencing is going to be like until we have that refactor in place and some of this will work itself out. A few things for both your and @philipbaileynar consideration:

1. While not discounting that "There is some effort required to create and maintain a new project type.", I want this question of whether or not to create a new project type or not to largely shift to one of who's paying and curating it. If there is an audience that wants this (and most importantly is paying for it), then we can shift this into a space of someone else's prerogative, we don't have to have the deck-chairs discussion.
2. When it comes to how these decisions impact our production grade tools, that does require more serious conversation. For me we have one end of the spectrum where projects (e.g. RS_Context and Channel_Area) are largely just inputs or context to other projects, and then a lot of downstream projects (e.g. TauDEM is downstream of RS_Context and Channel Area, VBET is downstream of TauDEM, and BRAT and Confinement are downstream of both of those). I really do not like inputs or intermediates in those downstream projects (e.g. hydrology and anthropogenic) that are not exposed more transparently as their own things, when they are used in multiple, subsequent, downstream projects. It creates a situation where bigger, more expensive, and more cumbersome projects like BRAT are being run when that is not what people are really after. It also makes it difficult to massage and transparently tweak those
3. This splitting and cross-referencing model will lead to leaner projects that are more focused on their own outputs and their own audience. It also isolates the pieces that users might modify through calibration or curation, in their own pieces. I believe this will make it easier to serve them when we get into the QRiS and RiS space. I could be wrong, but that's where I am coming from.

I know some would just like to chalk this up to Joe's splitting tendencies, but the best solution is clearly one where the warehouse refactor and architecture makes these type of decisions up to the prerogative of the developer and not one that causes the system administrators undue heartburn.

[philipbaileynar]

I have realized that my resistance is based in the fact that, with the current version of the warehouse, North Arrow Research bears the very real financial burden (in both time and AWS costs) any time that the number of project types as well as the number of projects overall increases.

The new warehouse gives us the opportunity to transfer this financial burden from NAR to the people creating and owning the projects. My resistance goes away now that those organizations that are adding projects to the new warehouse are going to pay.

[philipbaileynar]

New anthro and hydro models exist thanks to @jtgilbert !