chriswhong commented 10 years ago

Develop a standard schema for civic.json to add non-github metadata to civic app projects.

The projects page can look for civic.json in the root of every repo to add things like:

-Tag for in-progress, alpha, beta, complete -Tag for needs (I need a UX dev, I need a rails guru, etc) -Tag for sector (transportation, housing, government, council)

Please respond to this issue with ideas for what should be included in civic.json

derekeder commented 10 years ago

Related conversation in Chicago

https://github.com/ryanbriones/civicneeds/issues/4 https://github.com/open-city/open-gov-hack-night/issues/3

chriswhong commented 10 years ago

Here's a first draft.

status should be from a set of allowed values: beta, on hold, production, etc. (Not sure what these should be) thumbnail is just a url to an image for use in the catalog. bornAt links this project to particular event. geography defines a city, county, or other geography that this project is meant to serve. needs is an array of need objects, text is arbitrary. types is an array of type objects, text is arbitrary. categories is an array of category objects, text is arbitrary.

Should we limit types, needs and categories to predefined set of values? Or, perhaps we should have a set list of authorized types, but allow for arbitrary types under some different key?

[ { "status": "production - ongoing", "thumbnailUrl": "http://civic.com/hack.png", "bornAt": "National Day of Civic Hacking 2014", "geography": "NYC", "needs": [
{"need": "frontend dev"}, {"need": "UX Help"}, {"need": "rails developer"} ], "types": [ {"type": "dataset"}, {"type": "mobile app"}, {"type": "web app"}, {"type": "data visualization"}, {"type": "policy document"} ], "categories": [ {"category": "transportaion"}, {"category": "public safety"}, {"category": "politics"}, {"category": "news"}, {"category": "housing"}, {"category": "education"}, {"category": "health"} ] } ]

chriswhong commented 10 years ago

Perhaps add

"socialMedia":{ "twitter":"@myCoolProject", "facebook":"http://www.facebook.com/myCoolProject" }

chriswhong commented 10 years ago

Incorporated a test version of civic.json into a few repos and got all the backend stuff working to look for this file and add it to the rest of a repo's data. Check it out, it's live now at projects.betanyc.us

derekeder commented 10 years ago

Saw the civic.json project and the demo implementation on projects.betanyc.us. Awesome to see you guys run with this.

I've been thinking about it a bit more and I'm not sure a civic.json file added to each project is the answer. Here's my thinking:

The main idea behind tracking a list of repos via Github is to remove the need for curation and maintenance of these civic projects. Project directory sites in the past have quickly gotten stale and less useful shortly after the initial round of curation is done. I think a JSON file that needs to be manually updated is better and de-centralized, but with it the work of curation still remains and and actually becomes less consistent.

So, the question I pose is: what information is missing from the Github API that will help us better organize and collaborate on our civic projects?

One theme I've seen with the spec lined out by @chriswhong is a place to ask for specific skilled help (designer, python developer, etc). From my experience, these needs change all the time and are often more nuanced than a simple "we need X." To this end, wouldn't it be better to leverage the issue tracker and perhaps come up with a framework for opening and closing needs? Something like "Need: home page design." That way, when the need is met, it can be closed more naturally via conversation & commits, rather than someone remembering to update a JSON file.

We could also automatically detect the license type by looking for LICENSE.txt. Things like location and categories may still need a civic.json or some other solution, but my gist here is to look at ways to hook in to existing functionality like the issue tracker and establish a set of Github standards that accomplish our goals and help spur on collaboration if followed.

Curious to get some thoughts on this from @fgregg @evz @jpvelez @fkh @benbalter @kfogel @GovInTrenches

jpvelez commented 10 years ago

+1 for Github practices that help get work done and produce documentation programmatically as a byproduct.

Scratching my head on how to do elegantly that for location, categories, and data sources though.

Data sources = url of data or identified you used to build the app or analysis. Will make it easier to connect datasets to projects.

On Mon, Jan 20, 2014 at 3:33 PM, Derek Eder notifications@github.comwrote:

Saw the civic.json project https://github.com/BetaNYC/civic.json and the demo implementation on projects.betanyc.us. Awesome to see you guys run with this.

I've been thinking about it a bit more and I'm not sure a civic.json file added to each project is the answer. Here's my thinking:

The main idea behind tracking a list of repos via Github is to remove the need for curation and maintenance of these civic projects. Project directory sites http://commons.codeforamerica.org/ in the past have quickly gotten stale and less useful shortly after the initial round of curation is done. I think a JSON file that needs to be manually updated is better and de-centralized, but with it the work of curation still remains and and actually becomes less consistent.

So, the question I pose is: what information is missing from the Github API that will help us better organize and collaborate on our civic projects?

One theme I've seen with the spec lined out by @chriswhonghttps://github.com/chriswhongis a place to ask for specific skilled help (designer, python developer, etc). From my experience, these needs change all the time and are often more nuanced than a simple "we need X." To this end, wouldn't it be better to leverage the issue tracker and perhaps come up with a framework for opening and closing needs? Something like "Need: home page design." That way, when the need is met, it can be closed more naturally via conversation & commits, rather than someone remembering to update a JSON file.

We could also automatically detect the license type by looking for LICENSE.txt. Things like location and categories may still need a civic.json or some other solution, but my gist here is to look at ways to hook in to existing functionality like the issue tracker and establish a set of Github standards that accomplish our goals and help spur on collaboration if followed.

Curious to get some thoughts on this from @fgregghttps://github.com/fgregg @evz https://github.com/evz @jpvelez https://github.com/jpvelez @fkhhttps://github.com/fkh @benbalter https://github.com/benbalter @kfogelhttps://github.com/kfogel @GovInTrenches https://github.com/GovInTrenches

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-32799153 .

jpvelez commented 10 years ago

In the spirit of having as few fields as possible, I think "bornAt" and "status" are probably not critical. Status can be inferred from recent commit history.

kfogel commented 10 years ago

Derek Eder notifications@github.com writes:

So, the question I pose is: what information is missing from the Github API that will help us better organize and collaborate on our civic projects?

Amen -- that's the right question, I think.

One theme I've seen with the spec lined out by @chriswhong is a place to ask for specific skilled help (designer, python developer, etc). From my experience, these needs change all the time and are often more nuanced than a simple "we need X." To this end, wouldn't it be better to leverage the issue tracker and perhaps come up with a framework for opening and closing needs? Something like "Need: home page design." That way, when the need is met, it can be closed more naturally via conversation & commits, rather than someone remembering to update a JSON file.

Using the GitHub issue tracker API for stuff like this seems like the right way to go.

We could also automatically detect the license type by looking for LICENSE.txt.

There are some projects out there already to automate license detection (since it's not always in LICENSE.txt, and even when it is it's not always formatted in exactly the same way from project to project). http://www.fossology.org is one such, but I think there are others too.

Things like location and categories may still need a civic.json or some other solution, but my gist here is to look at ways to hook in to existing functionality like the issue tracker and establish a set of Github standards that accomplish our goals and help spur on collaboration if followed.

+1. The GitHub API is a pretty rich place, and getting richer all the time as far as I can tell. Much better to invest time in using it to solve problems (even in cases where it currently does so imperfectly) than to create new systems for solving those problems (because the new systems would be imperfect too, and they'd be new).

-K

GovInTrenches commented 10 years ago

1) One of the things I'd like to do is to enable searching by subject. Once we're able to plug in civic.json into a UI for people who have never heard of github, it would be useful to be able to sort by subject matter. There's a big typography project that would need to happen to successfully categorize the wide world of projects (which @SmartChicago has been looking at) - but at minimum being able to help tag projects by things like "311", "bikes", "legislation" would be useful.

2) The other thing in terms of process that would be interesting would be to integrate this into different hosting services. Rather than host all these apps on our own servers, to have Civic Commons / CfA Commons be a hosting provider. (Like Smart Chicago is locally.) That would ensure that we would grab the right information when it gets built (since they would need to provide that as part of the deal) and it would mean somebody is always looking at the list. (We'd have to figure out exact logistics on that but it's one way to solve the problem of sites going stale)

3) Contact information - At minimum, I should be able to ping the technical person's email so that even if a non-technical person doesn't get anything else about what's going on - they can still reach out.

jpvelez commented 10 years ago

Yes, that's what we mean by "categories." The question is how to do this seamlessly, so people don't have to do extra work to enter them.
Not sure what you mean by "integrate this into different hosting services." CfA commons doesn't host apps, it just links to them.
We might be able to grab this from the "Contact Us" section of the README, if it's there. This is usually pretty standard.

GovInTrenches commented 10 years ago

Sorry, was responding to @derekeder via email and hadn't seen the previous post. Categories would almost need a whole separate discussion if we're doing pre-defined values anything beyond arbitrary. Perhaps we could start with 'tags' as a start-point and then add categories as pre-defined values.

The hosting idea hasn't happened yet, but I'm saying it would be a possible solution to making sure somebody is always updating the entries and at least checking to make sure the projects still exist.

jpvelez commented 10 years ago

Folksonomy -> taxonomy is a good idea, except it would be difficult to get people to update a bunch of disparate repos once we landed on a taxonomy.

If we were to define a taxonomy, the way to do it would be to amass several more civic tech project lists, and distill public policy areas (health, 311, transportation) and project types (analysis, app) from that.

fgregg commented 10 years ago

What uses does a taxonomy serve? What would be possible to do that is not possible to do now?

On Mon, Jan 20, 2014 at 5:58 PM, Juan-Pablo Velez notifications@github.comwrote:

Folksonomy -> taxonomy is a good idea, except it would be difficult to get people to update a bunch of disparate repos once we landed on a taxonomy.

If we were to define a taxonomy, the way to do it would be to amass several more civic tech project lists, and distill public policy areas (health, 311, transportation) and project types (analysis, app) from that.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-32809111 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

jpvelez commented 10 years ago

Finding projects that pertain to a policy area like 311/transit or type statistical analysis/infrastructure/civic app, and analyzing the breakdown projects according to these criteria (probably less important.)

Making things easier to find might lead to more project awareness / use / reuse. To be worth doing, it would have to be substantially more useful than search.

chriswhong commented 10 years ago

Agree, bornAt isn't really necessary, I just thought it might be neat to tie projects to events. This could just as easily be in the description and would still show up in searches based on the description.

My thoughts on status are that I would like some way to distinguish what is a finished product versus what is a work in progress. I think there are lots of projects that start and never get finished, which is fine, but it would be great to filter the finished projects to make a nicer "showcase" list of actual, working, show-able projects.

On Mon, Jan 20, 2014 at 4:45 PM, Juan-Pablo Velez notifications@github.comwrote:

In the spirit of having as few fields as possible, I think "bornAt" and "status" are probably not critical. Status can be inferred from recent commit history.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-32800036 .

chriswhong commented 10 years ago

My thoughts on categories are that they should be more tag-like for now, while the number of projects as small. Just like when tagging a blog post, add as many as come to mind and hope it ends up in someone's search/filter.

When we have hundred or thousands of data points that are all tagged, we can have a better discussion about what the "official" category values ought to be. Just my two cents.

-C

On Mon, Jan 20, 2014 at 7:03 PM, Juan-Pablo Velez notifications@github.comwrote:

Finding projects that pertain to a policy area like 311/transit or type statistical analysis/infrastructure/civic app, and analyzing the breakdown projects according to these criteria (probably less important.)

Making things easier to find might lead to more project awareness / use / reuse. To be worth doing, it would have to be substantially more useful than search.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-32809398 .

derekeder commented 10 years ago

Good stuff. It would be useful for feature selection to come up with some user stories for how someone would use this tool and for what purpose. I'd rather work towards actually engaging people rather than making a bunch of 'it would be cool if' features.

@tothebeat blogged about how he wasn't sure what project to help out on. I could see this list serving the need of "I'm a developer with X skill. What projects are out there that I could plug in to?"

Are there others?

ondrae commented 10 years ago

Hi everyone.

To second the @tothebeat request, at Code for America we get an immense amount of offers for volunteer help. With a civic.json standard and good Github Issue labels, we could easily build a tool that would filter projects on programming language, category, time commitment, and geography. With that we could offer up interesting projects for volunteers to work on. I found a rough prototype someone else built called Issuehub.io. The extra metadata adds a lot of value to that type of tool.

I could be convinced either way if it should be a file in each repo or one master list.

jpvelez commented 10 years ago

Andrew: @derekeder is working on that. Ping him.

derekeder commented 10 years ago

@evz and I made some changes to the hack night projects list to support a custom project-needs label.

Check it out: http://opengovhacknight.org/projects.html#/?search=project needs

The idea is if you need something for your project and you're looking for outside help, create a new issue describing it and add the project-needs label to it and it will show up on our list.

One thing that I'm not keen on about issuehub.io is it hooks in to all of Github and looks at some very general labels. I don't know about you guys, but just because I file an issue on Github doesn't mean I'm asking for someone else to fix it for me. More often than not, it's just a reminder for me to do it later.

We'll be showing this off at tomorrow's hack night. We'll see how people use it.

jpvelez commented 10 years ago

Awesome. Cheap feedback: "projects need" bit would look better under the project stats, in the right-column.

Juan-Pablo Velez 312-218-5448 jpvelez@gmail.com

On Mon, Jan 27, 2014 at 11:56 AM, Derek Eder notifications@github.comwrote:

@evz https://github.com/evz and I made some changeshttps://github.com/open-city/civic-json-worker/commit/545c24e82821152635b2571d9e75e21cafa7c8deto the hack night projects list to support a custom project-needs label.

Check it out: http://opengovhacknight.org/projects.html#/?search=projectneeds

The idea is if you need something for your project and you're looking for outside help, create a new issue describing it and add the _project-needs_label to it and it will show up on our list.

One thing that I'm not keen on about issuehub.io is it hooks in to all of Github and looks at some very general labels. I don't know about you guys, but just because I file an issue on Github doesn't mean I'm asking for someone else to fix it for me. More often than not, it's just a reminder for me to do it later.

We'll be showing this off at tomorrow's hack night. We'll see how people use it.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33401523 .

chriswhong commented 10 years ago

This is rad and that it's clickable and has an explanation of the need is extra rad.

-C

On Mon, Jan 27, 2014 at 1:25 PM, Juan-Pablo Velez notifications@github.comwrote:

Awesome. Cheap feedback: "projects need" bit would look better under the project stats, in the right-column.

Juan-Pablo Velez 312-218-5448 jpvelez@gmail.com

On Mon, Jan 27, 2014 at 11:56 AM, Derek Eder notifications@github.comwrote:

@evz https://github.com/evz and I made some changes< https://github.com/open-city/civic-json-worker/commit/545c24e82821152635b2571d9e75e21cafa7c8de>to the hack night projects list to support a custom project-needs label.

Check it out: http://opengovhacknight.org/projects.html#/?search=projectneeds

The idea is if you need something for your project and you're looking for outside help, create a new issue describing it and add the _project-needs_label to it and it will show up on our list.

One thing that I'm not keen on about issuehub.io is it hooks in to all of Github and looks at some very general labels. I don't know about you guys, but just because I file an issue on Github doesn't mean I'm asking for someone else to fix it for me. More often than not, it's just a reminder for me to do it later.

We'll be showing this off at tomorrow's hack night. We'll see how people use it.

— Reply to this email directly or view it on GitHub< https://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33401523>

.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33404450 .

ondrae commented 10 years ago

Thats great. I agree that a specifically asking for help can be really helpful and it fits into the UI of your project page nicely. NYC has it over on the right column pulling from the civic.json file. I'm with you that Github is a better place for that.

Another method I've used well is having beginner labels on an issues. Then I link to the list of beginner issueshttps://github.com/codeforamerica/bizfriendly-web/issues?labels=beginner+friendly&page=1&state=open, so people have a good place to get started as they are learning the project. Could that fit in as well, or is being linked to from the README enough?

Andrew Hyder :: Developer Relations Engineer 2013 Code For America Fellow @hackyourcity http://twitter.com/hackyourcity

On Mon, Jan 27, 2014 at 9:56 AM, Derek Eder notifications@github.comwrote:

@evz https://github.com/evz and I made some changeshttps://github.com/open-city/civic-json-worker/commit/545c24e82821152635b2571d9e75e21cafa7c8deto the hack night projects list to support a custom project-needs label.

Check it out: http://opengovhacknight.org/projects.html#/?search=projectneeds

The idea is if you need something for your project and you're looking for outside help, create a new issue describing it and add the _project-needs_label to it and it will show up on our list.

One thing that I'm not keen on about issuehub.io is it hooks in to all of Github and looks at some very general labels. I don't know about you guys, but just because I file an issue on Github doesn't mean I'm asking for someone else to fix it for me. More often than not, it's just a reminder for me to do it later.

We'll be showing this off at tomorrow's hack night. We'll see how people use it.

— Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33401523 .

jpvelez commented 10 years ago

Just realized you can move the projects over to the right hand size because those are all columns, so moved them below 'contributors' so things look less cramped: http://opengovhacknight.org/projects.html

details

sheldonrampton commented 10 years ago

I would suggest basing civic.json on Schema.org's existing SoftwareApplication schema definition:

http://schema.org/SoftwareApplication

chriswhong commented 10 years ago

Thanks! This is very interesting... probably overkill, but a great reference for describing a project.

On Tue, Jan 28, 2014 at 10:28 AM, sheldonrampton notifications@github.comwrote:

I would suggest basing civic.json on Schema.org's existing SoftwareApplication schema definition:

http://schema.org/SoftwareApplication

Reply to this email directly or view it on GitHubhttps://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33488536 .

noneck commented 10 years ago

Chriswhong, Sheldon makes a great point. We should be using an open stander.

ameensol commented 10 years ago

Hi everyone, just joining this discussion. You guys have put in a lot of great work over the last several months to make this a reality. I'm happy to contribute where I can.

It's possible to implement an open standard such as the one @sheldonrampton mentioned by using a JSON-LD context for our civic.json files. Here's an example:

{
  "@context": {
    "schema": "http://schema.org",
    "thumbnail": "schema:thumbnailUrl",
    "types": "schema:applicationCategory",
    "categories": "schema:keywords"
  },
  "thumbnail": "*url*",
  "types": [
    "Dataset",
    "Mobile App"
  ],
  "categories": [
    "Transportation",
    "Public Safety"
  ],
  "status": "alpha",
  "schema:Place": "nyc",
  "needs": [
    "frontend",
    "ux",
    "rails"
  ]
}

I couldn't find schema.org terms definitions for "status" and "needs", but it isn't out of the question to develop our own definitions for these - or any other civic.json terminology - to go alongside this standard. It would take some time, obviously, and may be a pre-optimization.

We also probably don't want to require each developer to have to add this context to each and every one of their civic.json files, especially when nearly all of it is going to be redundant (unless we want them all to learn about Linked Data, which wouldn't be horrible). One solution JSON-LD allows would be to use context as a url, like this:

{
  "@context": "*contextUrl*",
  "more fields": "..."
}

The contextUrl would then map to a webpage where our context lives. This still requires an additional field for the developer, so its worth exploring to see if we can do all the open standards bit on the backend and then just require a specific vocabulary: "needs", "categories", "types", etc... This would allow 0 change to the current civic.json format, and we'd still be using open standards. I think it would work as long as our json endpoint had the context object.

daguar commented 10 years ago

Getting into this late because there's interest in Oakland around this (holler.)

Putting on my run-of-the-mill Brigade member fedora, I'm thinking through what hooking up to this involves on the user end because it seems like minimizing friction is a big part of it.

It seems like the original thought was to place a civic.json file, but -- per @derekeder's comments here -- the direction is moving towards a hybrid of using GitHub Issues for needs, and a civic.json file for the rest?

(Apologies if my understanding's incorrect -- this discussion is spanning a bunch of repos, threads, and email chains.)

IMHO, I'd vote for doing less, and just sticking with civic.json. My rationale is:

Each additional thing you're asking end-users to do will reduce adoption + increase likelihood of data staleness/non-conformity.
The needs attribute outlined in the spec seem more like general roles for participation, rather than concrete tasks/features. Concomitantly, these needs probably won't change very often. That said, I know the Chicago mafia has a lot more experience with this, so tell me if I'm crazy.
If we're asking people to add a civic.json file already, it seems like it minimizes friction to just keep needs in there.

I totally get the desire to make use of GitHub's infrastructure for this, but because using Issues doesn't seem to obviate the need for most of the civic.json file, I'm worried about the additional layer of complexity this adds for not that much benefit.

Fundamentally it seems like it's more a substantive discussion of how we're defining "need": a concrete task to be done vs. a skill set. I worry about the former suffering from overspecification.

Anyway, just thinking through this as someone who'd love to implement it in my city/brigade, and trying to sort out what exactly it will entail.

cc @migurski @ondrae for your thoughts, given the work you're doing on civic-json-worker

jpvelez commented 10 years ago

@daguar the best entry point is to read this overview of civic-json-worker and civic.json.

the overall project roadmap is here.

tl;dr: the goal is to describe projects with as little data entry burden as possible. that means:

get most project fields from Github API
reach a consensus on a very short list of additional - and likely optional fields - project topic (311, transit), type (web app, data analysis)
figure out how to make these fields as easy to gather as humanly possible. that means either tease them out from useful github issues (i.e. getting project need from repo issue that also helps describe the need) or make find clever ways to make the data entry easy (autocomplete tags on the project page, drop in fields into a github issue so non-repo-owners can document projects, etc.)

And how's this being carried out?

Brigade project pages simply submit github repo urls to civic-json-worker, a project tracking API that goes out and collects all the fields from github. the project page then pull the complete project data to display on the page. @chriswhong is writing a blog post to help existing brigade website read/write to the API from their existing pages, and is probably going to turn betanyc's project page into a generic project page for new brigades to use.
civic-json-worker needs to be adapted to accept and serve projects from multiple cities. right now it only does it for Chicago (we decided to build on the initial flask version of the API, not the NYC node version. NYC's project page will swap out their backend once that happens.) @ondrae and @evz are leading this, the work is happening on CfA's new fork of civic-json-worker.
civic.json started off as a file you dropped into your repo to document everything. but this concept is limited because you can get many of the fields from the Github API, and it puts the burden on the repo owner to fill in / maintain the additional fields. we already experimented with this in Chicago, and it didn't work. better to have the fields be a byproduct of issue tracker activity and/or to make it easy for non-repo-owners to maintain the fields. so now civic.json is going to be programmatic output of civic-json-worker. we make robots build them, instead of humans, and we pass them to the projects pages for display. @chriswhong and myself are organizing the discussion around what fields should be in the spec, and what are some efficient ways to make robots build them; work is going on in this repo.

migurski commented 10 years ago

Thanks JP & Dave, this is a super useful discussion and I think we’re honing in on an incredibly useful community resource here.

FWIW, I am somewhat skeptical of the idea that meaningful needs and project description will simply fall out of existing Github features. CfA’s desire here is to have a high-quality, primary source of project-related data. Github will help with that, but it can’t be the sole source of information. I view Github metadata as one set of data that we collect to build up a database, but I want to be sure that we’re open to alternative sources: sites other than Github, projects other than code, and so forth.

I’ve posted some related comments on codeforamerica/civic-json-worker issue 14 and 13.

jpvelez commented 10 years ago

@migurski No argument there. "Get everything from Github" was just a first-stab proxy for "make data gathering as easy as humanly possible because otherwise it will fail. Get everything you can through robots, make the rest stupid simple."

@chriswhong Can you close this issue so we can official move the discussion to the civic-json-worker and civic.json repos?

chriswhong commented 10 years ago

Donezo

BetaNYC / projects-list

civic.json for metadata #5

details