research the civic platform from hack oregon

[markdav-is]

Hack Oregon and Eugene have been chatting at a policy level about being a hack-oregon node and maybe the first non-pdx city to use civic platform http://www.hackoregon.org/civic-projects

I hung out with these folks as OSCON 2018 and we chat a few times a year. The "platform" is really more of set of react component to do fancy visualizations on the front end and an API on the back. Not sure about the tech there, but each dataset gets it's own ETL and that's pretty custom.

maps in the sandbox as well: http://civicplatform.org/sandbox using mapbox.

it would be fun to keep those conversations going with our civic projects and leverage some of the work they are doing as well.

[csjoblom]

Civic Platform is expensive and unless we have significant resources locally with that organization I don't forsee this happening. Based on conversations with Cat in the past this doesn't seem aligned with what they are doing.

[ryanrolds]

Right now we need to focus on completing and unblocking work along the path to deploying a Lane County version of the software. Not opposed, just wanting to make sure we prioritize our work for success at our immediate milestones.

@nohorse What benefits does Hack Oregon being to this project? Are there any risks (sunk time, costs, etc...) related to engaging with Hack Oregon? How do benefits to the project out weigh the risks?

[ryanrolds]

The code base does include being able to download a data file (CSV I think). There is nothing stopping Hack Oregon and Civic Platform from downloading and ingesting that data on their own.

[markdav-is]

I do remember the cost associated now that you mention it @csjoblom . I was origionaly just thinking that thier stuff looked pretty nice and that it was open-source. let me touch base with them and see if ti's a buy-in to "the platform" or if it's also a public repo we could just grab bits from.

[markdav-is]

this is the front end wiht the MIT license: https://github.com/hackoregon/civic

[ryanrolds]

@nohorse How do you envision it helping this project? A replacement frontend? Is it something we host or push data to them?

[markdav-is]

He're the back-end example project: https://github.com/hackoregon/backend-examplar-2018

[ryanrolds]

I'm looking at the example on http://civicplatform.org/sandbox and they appear to be serving much smaller datasets then we will be working with and are using the same techniques (giant JSON files containing boundaries) that our current codebase uses that are unsuitable for the volume of data and number of aggregations we are working with.

[ryanrolds]

tl;dr Civic Platform has the same issue identified in the Big Rocks document. We need solutions that solve the scaling issues that we've identified.

[markdav-is]

@ryanrolds I see this being an option for build good looking front-end interactive pretty data elements. It's cleanly separated into front-end JS and back-end dockerized API. I'm not personally a django fan, so I'm not pushing any technology choices.

mostly I just wanted to let this team know that there are pretty and free react components for the UI as well as pre-build open-data APIs make by a local civic non-profit org that we're having some strategy conversations with. So using these might unlock some additional resources.

I never really considered that we'd be hosting anything with them or buying into any sort of services, just leveraging OSS components from some folks we are friendly with.

I'm not driving this project, but will definitely share my knowledge of connections and energy that have been going into other OSS projects that might benefit this one too.

[ryanrolds]

My concern is that we have known scaling issues, issues that other projects trying to solve the same issue are encountering too. Suggesting solutions without first ensuring that it solves problems the project is facing distracts people and quickly becomes a game of unvetted idea whack-a-mole. Have you read https://paper.dropbox.com/doc/SpeedUpAmerica-Big-Rocks--AbuTOwBbY61nfxgmk7JmE9W6Ag-c1sMe5n0iXr2fkc7d3K0T and are you familiar with the scaling issues we are trying to address?

[markdav-is]

This research issue was not in response to the Big Rocks Doc. It was in response to the repo we forked being kinda ugly and clunky. If @ryanrolds and @csjoblom don't want to explore this, just close the ticket. No worries my friends. I'm just here to help. 🦄

[ryanrolds]

Everyone is in agreement that the codebase we inherited is clunky, but any solution we develop will have to consider the performance issues outlined in the Big Rocks document. The point of working with the existing codebase is two part:

1. It should handle a Lane County sized dataset, enabling us to hit our first milestone with minimal effort
2. We take what we learn from working with it and use it develop something more performant

We've identified bottlenecks that will need to be addressed and those items are documented in the Big Rocks document. Any visualization tools, datastores, languages we use will all have to address the data and visualization issues outlined in the Big Rocks document. It's critical that all ideas be vetted against those "knowns" and as we uncover additional issues we will add them to that document.

Your input is appreciated. What I'm requesting is that everyone on the project be mindful of the problems this project faces and when solutions are proposed. It's also helpful if the suggestion clearly states the problem it's trying to address and how it relates to the project.

By suggesting solutions without vetting them against our "knowns" you're asking that someone else on the team take time and vet your off the cuff suggestion. At the moment, we need people taking and completing tickets that move us towards our Lane County milestone. If you have ideas that help us reach that milestone please suggest them, but I do ask that you measure the idea against our long term goals and "knowns" first.