311 Complaint Type Buckets

[fma2]

Resources: https://data.cityofnewyork.us/Social-Services/311-Complaint-Types/h4xh-jcuz http://www1.nyc.gov/311/index.page

[fma2]

Some documentation for more specific 311 complaints' buckets-- 1) Figure out what 311 complaint type you want to capture.
2) Copy and paste the lib/spy_glass/registry/nyc-311.rb into a new ruby file. Save it in the registry folder (same folder as lib/spy_glass/registry/nyc-311.rb).

3) In the new file, add the complaint types to the opts hash as a part of the value for source.

4) Run bundle exec rackup in command line. 5) Open http://localhost:9292/ in the browser. You should see your feed linked there.

[fma2]

@jmcole6539 @interglobalmedia, sharing the above for any buckets you want to create with the 311 data

[interglobalmedia]

@jmcole6539 @fma2 thanks farheen for sharing! I will look at the various issues listed in the 311 data set(s) and see if there is anything I might be interested in. Thanks! @jmcole6539, I came across a lot of cool data which would be of definite interest to you. When I go into the various sources before we all meet on Wednesday, I'll put together a list of things with the links for you to look at, so you can access them quickly. And those definitely are "event based".

Maria

[fma2]

Pushed the above to develop branch: https://github.com/BetaNYC/citygram-services-nyc/blob/develop/lib/spy_glass/registry/nyc-311-heating-complaints.rb and deployed to staging-citygram-services-nyc.herokuapp.com.

@dirkkelly, do you want to check if it works on Citygram staging environment?

Citygram::DatabaseHelper.database[:publishers].insert(
  visible: true,
  active: true,
  title: '311 Heat/Hot Water Complaints',
  endpoint: 'https://staging-citygram-services-nyc.herokuapp.com/nyc-311-heating-complaints',
  city: 'new-york',
  icon: 'cone.png'
)

I used cone.png because I didn't see an icon for housing/heat/hot water in the citygram-nyc repo

[fma2]

Current 311 buckets in progress --

• [ ] Housing/Quality of Life Bucket (Issue https://github.com/BetaNYC/citygram-services-nyc/issues/6)

[jmcole6539]

@fma2 thanks so much for that code - I was going to request that going forward for testing it.

@interglobalmedia @fma2 Thanks for sending that json feed! I spent a good part of yesterday going through every iteration on the http://www1.nyc.gov/311/index.page and "mapping" the prompts on that page to the Complaint Type that they ultimately are tagged as. Hopefully this information will be helpful not only for the QoL bucket but also Health, etc. I think that your scrape achieves the exact same process. I need to learn how to scrape! But it was interesting to do this and I found some inconsistencies, which seems to be uncovered in your json as well.

I'll take a look more into this tonight - unfortunately I won't be able to make it on Wednesday because I have a Board meeting, but would it be possible to meet sometime later in the week/this weekend? If not, I'll definitely be at next week's meetup, which I know is in Queens so may be out of the way for you guys.

[fma2]

@jmcole6539, you're welcome!

Also, if you haven't seen the other json in that repo, it contains topics, subtopics, subtopics' json links, and added details about each subtopic found in its unique json: categories, complaint types, and topic names": https://github.com/fma2/311-make-a-complaint-page/blob/master/public/topics-subtopics-selected-types.json.

Check out the repo's README for a little more context.

No worries about Wednesday. We can just continue the conversation here and pick up wherever we are at next week. Does that work?

[fma2]

I almost forgot to add to the above that, for testing, get a SOCRATA_APP_TOKEN and place it in the .env file.

You can get one by creating an account on the Open Data portal: https://data.cityofnewyork.us

[fma2]

@jmcole6539, check this page out: http://www1.nyc.gov/apps/311srmap/

On the right-hand panel, select a "Service Request Category". Then, select "Advanced Search" and look through the list of complaint types in the 'Complaint Type' drop down.

It's even better than what I did! I'm manually searching some of those on the open data complaint types list (I linked a json of it, as I find its much easier to cmd-F or ctrl-F on it than here), and they are checking out--I'm finding all of the ones I'm searching. Haven't figured out the best way to scrape it though.

[jmcole6539]

@fma2 wow - this is great! what i've been doing is mapping each of the various methods of submitting/reviewing (mobile app, 311 website, etc.) in the same Excel spreadsheet (what do coders usually use to share tabular data like this - Google docs?) so that we can also see the tagging inconsistencies across platforms, as well as to get a picture of the full universe of how they've been tagged to date.

I'll add this "bucketing" information to that list as well; hopefully it will find a "home" for the "homeless" complaint types that i couldn't trace back to figure out how a user could complain about them on the mobile/web form.

[dirkkelly]

@fma2 sorry been a busy week, I will try to look at this tonight and will add to the 311 environment.

[dirkkelly]

@fma2 Added your feed to https://citygram-nyc-311.herokuapp.com

heroku run rake console -a citygram-nyc-311

require_relative 'lib/database_helper'

Citygram::DatabaseHelper.database[:publishers].insert(
  visible: true,
  active: true,
  title: '311 Heat/Hot Water Complaints',
  endpoint: 'https://staging-citygram-services-nyc.herokuapp.com/nyc-311-heating-complaints',
  city: 'new-york',
  icon: 'cone.png'
)

heroku run rake publishers:update -a citygram-nyc-311

heroku ps -a citygram-nyc-311
> === web (1X): `bundle exec unicorn -c ./config/unicorn.rb`
> web.1: up 2015/03/12 22:40:20 (~ 1m ago)

heroku ps:scale worker=1 -a citygram-nyc-311
> Scaling dynos... done, now running worker at 1:1X.

Looking good, I think to get this to production all we need is

• [ ] icon
• [ ] better fitting title

Thank you so much for this, sorry it took me so long to get it online!

[fma2]

Awesome! So, the heat and hot water notifications feed was just a sample, so I'm glad to see its working.

Relevant update: @noneck connected me with someone from 311 who I'm supposed to meet with soon to get a better idea of what 311 complaints types fit under which category (Housing and Development, Education, Environment, Health, Civic Services, Business, Transportation, Culture and Recreation, Social Services, Public Safety). Once that is clear, I think making a feed for each category will be super easy, especially now that we're sure this heat/hot water sample feed works!

[dirkkelly]

@fma2 that's excellent news, great work getting your mind around all this data, those categories look like winners, looking forward to them.

[jmcole6539]

@fma2 yes those categories look great. it looks like they match the existing grouping of NYC's agencies here (which i'm sure was the plan): http://www1.nyc.gov/nyc-resources/agencies.page. also shared this with @noneck at Hack Night tonight and we found they match (mostly) the categories on the Open Data portal as well.

I first thought that we'd be able to use "Agency Contacted" to map the 311 request to the category, but it looks like at least some agencies are "tagged" as multiple categories (NYC Office of the Actuary is in Business and Civic Services, for example). Not sure if these agencies are the same ones that would have 311 requests though...are you doing this analysis as well? Don't want to duplicate efforts.

[fma2]

@jmcole6539 thanks for cross-checking with the open data portal and the list of NYC agencies!

Yes, 'Agency Contacted' tagged to multiple categories sounds about right, and I'm not doing that analysis. Before you do that analysis, when creating the query for each category for the new feeds, do we want to include "Agency Contacted" as a parameter in addition to "Complaint Type"? And should I ask the individual who works at 311 about 'Agency Contacted' as well? I'm scheduled to meet with her on Monday.

[fma2]

I've listed all of the categories and complaint types I scraped from the NYC 311 Booker App and the NYC 311 Service Request map here: https://docs.google.com/document/d/1ON143VSMC9bDExlWFV7sG-Ly29949JAg5QPLkr2DGpk/edit#heading=h.1yecoljqe26q

[interglobalmedia]

@fma2 There actually is something that I might want to create. It has recently come to my attention that there is a big sex offender problem in my area. They reside in and hang around old Bellevue in the men's homeless shelter. I don't know if this is feasible, but is it possible to create a complaint bucket for this particular issue in this particular neighborhood? I notice that people in my neighborhood have lodged "live" complaints on 311. Not of this nature, but generally yes. One of my neighbors went to some meeting last night regarding this, and ended up on TV! Sorry I missed it. Anyway, let me know. I feel very strongly about this.

[interglobalmedia]

@fma2 I guess it would fall under public safety, right?

[fma2]

@hhassan1230 check this issue out

[jmcole6539]

@fma2 hi Farheem! I know it's been a long time, but i've started a Masters at Carnegie Mellon in Pittsburgh and working with the CFA brigade here to bring Citygrams to Pittsburgh. We'll be looking over the wiki that CFA has provided but just wanted to let you know. The NYC citygrams is looking great!