Closed derekeder closed 10 years ago
jcgiuffrida
commented
10 years ago This is unexpected - we were planning to add dozens more from here as soon as the ETL process is ready to accept them. I might be misreading this, so let's discuss on the call tomorrow. The data used in the example queries and the data that Plenario has should be thought of separately: we want to show off the coolest stuff, but the point has always been that we would also have tons and tons of data.
cecat
commented
10 years ago Does pairing down improve performance or are we wanting visual simplicity in the interface (vs long list ). If the latter, I think the long list is a bonus and we point out one of the things we will work on in year 2 is better data set navigation.
I was gonna ask if we could get the 311 data for water issues - we are starting a project related to urban flooding with Center for NHood Tech.
CeC
On Sep 15, 2014, at 9:18 PM, Jonathan Giuffrida notifications@github.com wrote:
This is unexpected - we were planning to add dozens more from here as soon as the ETL process is ready to accept them. I might be misreading this, so let's discuss on the call tomorrow. The data used in the example queries and the data that Plenario has should be thought of separately: we want to show off the coolest stuff, but the point has always been that we would also have tons and tons of data.
— Reply to this email directly or view it on GitHub.
fgregg
commented
10 years ago City hasn't released the 311 flooding calls. @tomschenkjr said he hoped it was going to be released this year.
cecat
commented
10 years ago Ah. It might have property value implications so I wonder if they will somehow blur the locations.
CeC
On Sep 15, 2014, at 9:34 PM, Forest Gregg notifications@github.com wrote:
City hasn't released the 311 flooding calls. @tomschenkjr said he hoped it was going to be released this year.
— Reply to this email directly or view it on GitHub.
derekeder
commented
10 years ago I'm concerned mostly about presentation on this one. I'm ok with a long list (to @cecat's point), but I want to make sure that the data is loaded properly and the queries actually work when people query them.
I'd like to focus on the datasets essential to #93 first and then swing around and add more once those are solid. @lucaluca let's go over this in the meeting today.
jcgiuffrida
commented
10 years ago Here are the eight datasets we would like for the queries in https://github.com/datamade/plenario/issues/93:
| Portal | Human name | url | rows | unique id (logical key) | observation date | location | update frequency |
|---|---|---|---|---|---|---|---|
| San Francisco | Mobile Food Schedule | http://data.sfgov.org/resource/jjew-r69b | 55562 | scheduleid | Addr_Date_Create | lat-long | daily |
| San Francisco | Graffiti - 30 Days | http://data.sfgov.org/resource/p6sg-7yp7 | 2060 | CASE ID | OPEN DATE | POSITION | daily |
| San Francisco | Case Data from San Francisco 311 (SF311) | http://data.sfgov.org/resource/vw6y-z8j6 | 1044942 | CaseID | Opened | Point | daily |
| San Francisco | Crimes 2003-present | https://www.dropbox.com/s/0pd1y0sjk5dwrzb/master.csv?dl=1 | IncidntNum | DateTime | lat-long (Y-X) | none | |
| New York City | 311 Service Requests from 2010 to Present | http://nycopendata.socrata.com/resource/erm2-nwe9 | 7947652 | Unique Key | Created Date | lat/long | Daily |
| New York City | NYPD Motor Vehicle Collisions | http://nycopendata.socrata.com/resource/h9gi-nx95 | 390583 | Unique Key | DATE | lat/long | Daily |
| Boston | Mayors 24 Hour Hotline (NEMO Snow Related Calls) | http://data.cityofboston.gov/resource/r3qt-vrtj | 13315 | Case Enquiry ID | open_dt | Geocoded_Location | Yearly |
| Austin | 311 Unified Data | http://data.austintexas.gov/resource/i26j-ai4z | 88533 | Service Request Number | Created Date | (Latitude.Longitude) | Daily |
evz
commented
10 years ago @lucaluca There's no header row on the San Francisco crime file that you link to above. I went ahead and made one that has a header row and while I was at it, merged the date and time column. It's here:
http://plenario-dev.s3.amazonaws.com/sfpd_incident_all_datetime.csv
derekeder
commented
10 years ago To keep things manageable, we're going with the 35 we had already plus the 8 listed above. Here's the final list of 43:
| Dataset | Attribution |
|---|---|
| 311 Service Requests - Abandoned Vehicles | City of Chicago |
| 311 Service Requests - Alley Lights Out | City of Chicago |
| 311 Service Requests - Garbage Carts | City of Chicago |
| 311 Service Requests - Graffiti Removal | City of Chicago |
| 311 Service Requests - Pot Holes Reported | City of Chicago |
| 311 Service Requests - Rodent Baiting | City of Chicago |
| 311 Service Requests - Sanitation Code Complaints | City of Chicago |
| 311 Service Requests - Street Lights - All Out | City of Chicago |
| 311 Service Requests - Street Lights - One Out | City of Chicago |
| 311 Service Requests - Tree Debris | City of Chicago |
| 311 Service Requests - Tree Trims | City of Chicago |
| 311 Service Requests - Vacant and Abandoned Buildings Reported | City of Chicago |
| 311 Service Requests from 2010 to Present | 311 |
| 311 Unified Data | None |
| Alternative Fuel Locations | None |
| Building Permits | City of Chicago |
| Business Licenses | City of Chicago |
| Case Data from San Francisco 311 (SF311) | San Francisco 311 |
| CDPH Environmental Complaints | Chicago Department of Public Health |
| CDPH Environmental Hold on City-Issued Permits and LUST NFR | Chicago Department of Public Health |
| CDPH Environmental Inspections | Chicago Department of Public Health |
| CDPH Environmental Permits | None |
| Cook County Recorder of Deeds - Foreclosures - 2011 Complete! | None |
| Cook County Recorder of Deeds - Foreclosures - 2012 January 1 to August 9 | Cook County Recorder of Deeds |
| Cook County Recorder of Deeds - Mortgages - 2011 Complete! | None |
| Cook County Recorder of Deeds - Mortgages - 2012 January 1 to August 9 | None |
| Cook County Recorder of Deeds - Quit Claim Deeds - 2011 Complete! | None |
| Cook County Recorder of Deeds - Quit Claim Deeds - 2012 January 1 to August 9 | Cook County Recorder of Deeds |
| Crimes - 2001 to present | Chicago Police Department |
| CTA - Ridership - Avg. Weekday Bus Stop Boardings in October 2012 | None |
| Disabled Parking | SFMTA, SFpark |
| Environmental Control - Asbestos Abatement Permits - 2012 Q3 | None |
| Environmental Control - Asbestos Demolition Permits - 2012 Q3 | None |
| Food Inspections | City of Chicago |
| Graffiti - 30 Days | San Francisco 311 |
| Mayors 24 Hour Hotline (NEMO Snow Related Calls) | Mayor's Office - 24 Hour Hotline |
| Micro-Market Recovery Program - Cases | City of Chicago |
| Micro-Market Recovery Program - Permits | City of Chicago |
| Micro-Market Recovery Program - Violations and Inspections | City of Chicago |
| Mobile Food Schedule | Department of Public Works |
| NYPD Motor Vehicle Collisions | Police Department (NYPD) |
| Red Light Camera Tickets | Chicago Tribune |
| sfpd_incident_all_datetime.csv | San Francisco Police Department |
jcgiuffrida
commented
10 years ago Will there be time to add any federal data or any Socrata datasets in the google doc? Even just the small ones.
On Sep 18, 2014, at 11:49 AM, Derek Eder notifications@github.com wrote:
To keep things manageable, we're going with the 35 we had already plus the 8 listed above. Here's the final list of 43: 311 Service Requests - Abandoned Vehicles City of Chicago 311 Service Requests - Alley Lights Out City of Chicago 311 Service Requests - Garbage Carts City of Chicago 311 Service Requests - Graffiti Removal City of Chicago 311 Service Requests - Pot Holes Reported City of Chicago 311 Service Requests - Rodent Baiting City of Chicago 311 Service Requests - Sanitation Code Complaints City of Chicago 311 Service Requests - Street Lights - All Out City of Chicago 311 Service Requests - Street Lights - One Out City of Chicago 311 Service Requests - Tree Debris City of Chicago 311 Service Requests - Tree Trims City of Chicago 311 Service Requests - Vacant and Abandoned Buildings Reported City of Chicago 311 Service Requests from 2010 to Present 311 311 Unified Data None Alternative Fuel Locations None Building Permits City of Chicago Business Licenses City of Chicago Case Data from San Francisco 311 (SF311) San Francisco 311 CDPH Environmental Complaints Chicago Department of Public Health CDPH Environmental Hold on City-Issued Permits and LUST NFR Chicago Department of Public Health CDPH Environmental Inspections Chicago Department of Public Health CDPH Environmental Permits None Cook County Recorder of Deeds - Foreclosures - 2011 Complete! None Cook County Recorder of Deeds - Foreclosures - 2012 January 1 to August 9 Cook County Recorder of Deeds Cook County Recorder of Deeds - Mortgages - 2011 Complete! None Cook County Recorder of Deeds - Mortgages - 2012 January 1 to August 9 None Cook County Recorder of Deeds - Quit Claim Deeds - 2011 Complete! None Cook County Recorder of Deeds - Quit Claim Deeds - 2012 January 1 to August 9 Cook County Recorder of Deeds Crimes - 2001 to present Chicago Police Department CTA
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56068222.
brettjgoldstein
commented
10 years ago I need to be able to say:
-we can tell the story of a place with city, county, state and federal -we have socrata, ckan and whatever
Brett Goldstein Senior Fellow in Urban Science Harris School of Public Policy University of Chicago @bjgol
On September 18, 2014 at 12:23:56 PM, Jonathan Giuffrida (notifications@github.commailto:notifications@github.com) wrote:
Will there be time to add any federal data or any Socrata datasets in the google doc? Even just the small ones.
On Sep 18, 2014, at 11:49 AM, Derek Eder notifications@github.com wrote:
To keep things manageable, we're going with the 35 we had already plus the 8 listed above. Here's the final list of 43: 311 Service Requests - Abandoned Vehicles City of Chicago 311 Service Requests - Alley Lights Out City of Chicago 311 Service Requests - Garbage Carts City of Chicago 311 Service Requests - Graffiti Removal City of Chicago 311 Service Requests - Pot Holes Reported City of Chicago 311 Service Requests - Rodent Baiting City of Chicago 311 Service Requests - Sanitation Code Complaints City of Chicago 311 Service Requests - Street Lights - All Out City of Chicago 311 Service Requests - Street Lights - One Out City of Chicago 311 Service Requests - Tree Debris City of Chicago 311 Service Requests - Tree Trims City of Chicago 311 Service Requests - Vacant and Abandoned Buildings Reported City of Chicago 311 Service Requests from 2010 to Present 311 311 Unified Data None Alternative Fuel Locations None Building Permits City of Chicago Business Licenses City of Chicago Case Data from San Francisco 311 (SF311) San Francisco 311 CDPH Environmental Complaints Chicago Department of Public Health CDPH Environmental Hold on City-Issued Permits and LUST NFR Chicago Department of Public Health CDPH Environmental Inspections Chicago Department of Public Health CDPH Environmental Permits None Cook County Recorder of Deeds - Foreclosures - 2011 Complete! None Cook County Recorder of Deeds - Foreclosures - 2012 January 1 to August 9 Cook County Recorder of Deeds Cook County Recorder of Deeds - Mortgages - 2011 Complete! None Cook County Recorder of Deeds - Mortgages - 2012 January 1 to August 9 None Cook County Recorder of Deeds - Quit Claim Deeds - 2011 Complete! None Cook County Recorder of Deeds - Quit Claim Deeds - 2012 January 1 to August 9 Cook County Recorder of Deeds Crimes - 2001 to present Chicago Police Department CTA
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56068222.
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56072919.
derekeder
commented
10 years ago @brettjgoldstein Re: we can tell the story of a place with city, county, state and federal In Chicago, we have City, County and NOAA data, which is Federal. Not sure what data from https://data.illinois.gov/ fits our requirements of unique id, datestamp, lat/long
Re: we have socrata, ckan and whatever We have Socrata and CSV. http://catalog.data.gov/dataset is CKAN if you can find something that fits our requirements
brettjgoldstein
commented
10 years ago JG — pls help. also look at fed data.gov
Brett Goldstein Senior Fellow in Urban Science Harris School of Public Policy University of Chicago @bjgol
On September 18, 2014 at 12:48:32 PM, Derek Eder (notifications@github.commailto:notifications@github.com) wrote:
@brettjgoldsteinhttps://github.com/brettjgoldstein Re: we can tell the story of a place with city, county, state and federal In Chicago, we have City, County and NOAA data, which is Federal. Not sure what data from https://data.illinois.gov/ fits our requirements of unique id, datestamp, lat/long
Re: we have socrata, ckan and whatever We have Socrata and CSV. http://catalog.data.gov/dataset is CKAN if you can find something that fits our requirements
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56076265.
derekeder
commented
10 years ago @brettjgoldstein @lucaluca we're also looking in to IL and Fed data to add.
jcgiuffrida
commented
10 years ago I have a list of Illinois datasets - I will put up a list of those meeting our criteria. If you could look into data.gov that would help. I found a few cool ones with rail disasters and maritime disasters but didn't try inserting.
On Sep 18, 2014, at 1:12 PM, Derek Eder notifications@github.com wrote:
@brettjgoldstein https://github.com/brettjgoldstein @lucaluca https://github.com/lucaluca we're also looking in to IL and Fed data to add.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56079546.
meemking
commented
10 years ago I will help look at federal rn
On Thu, Sep 18, 2014 at 1:21 PM, Jonathan Giuffrida < notifications@github.com> wrote:
I have a list of Illinois datasets - I will put up a list of those meeting our criteria. If you could look into data.gov that would help. I found a few cool ones with rail disasters and maritime disasters but didn't try inserting.
On Sep 18, 2014, at 1:12 PM, Derek Eder notifications@github.com wrote:
@brettjgoldstein https://github.com/brettjgoldstein @lucaluca https://github.com/lucaluca we're also looking in to IL and Fed data to add.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56079546.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56080720.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389
jcgiuffrida
commented
10 years ago Some good Illinois datasets are available in the google doc. More on their way, but we can definitely start with building permits and graffiti.
Note: all of these seem to stop in Jan 2013. Anyone know why?
meemking
commented
10 years ago I know we want to be realistic in how many datasets are added before launch, so here are two interesting federal data sets that dovetail off of the queries and topics we have covered so far but add breadth to Plenario in that they cover stuff (1) outside of Chicago and (2) non-Socrata based and (3) specifically sourced from federal databases.
Federal Car Accident Data http://catalog.data.gov/dataset/accidents-fars-national - (1975-current): contains information about crash characteristics and environmental conditions at the time of the crash. There is one record per crash. Extends our data here byeond NYC to highways. interesting to test correlations with weather.
Average monthly Retail Price of Electricity by state (links to EIA's API) http://www.eia.gov/beta/api/qb.cfm?category=1012 This TS would be interesting especially since its correlates with weather, storms, and impact in cities during storms. Statewide so makes it pretty easy to add, only varies in time.
If you want, I can find more or different ones, LMK.
On Thu, Sep 18, 2014 at 2:17 PM, Jonathan Giuffrida < notifications@github.com> wrote:
Some good Illinois datasets are available in the google doc https://docs.google.com/spreadsheet/ccc?key=0Au-2OHnpwhGTdGJzUWJ2SERwVXZLeDU4Y3laWFJvNEE#gid=0. More on their way, but we can definitely start with building permits and graffiti.
Note: all of these seem to stop in Jan 2013. Anyone know why?
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56088215.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389
cecat
commented
10 years ago Everyone....
Just an advance THANK YOU for the hard work you are doing to prep for what should be an exciting launch with Brett's talk next week. I am in DC today in in two separate gatherings the concept of Plenar.io was really excitedly received. One was a group of folks from OSTP, State Dept, a few companies, MacArthur Foundation, NIST, etc. talking about instrumenting cities to understand climate change. The other was a room full of 200 people from academic supercomputer centers, but more importantly a bunch of folks from NSF who were delighted to hear that a couple of relatively modest awards could have such impact.
I propose that after Brett returns from SFO we hold a pizza/beer celebration of year 1 of this 2 year grant.
CeC
On Sep 18, 2014, at 4:06 PM, meemking notifications@github.com wrote:
I know we want to be realistic in how many datasets are added before launch, so here are two interesting federal data sets that dovetail off of the queries and topics we have covered so far but add breadth to Plenario in that they cover stuff (1) outside of Chicago and (2) non-Socrata based and (3) specifically sourced from federal databases. \
Federal Car Accident Data http://catalog.data.gov/dataset/accidents-fars-national - (1975-current): contains information about crash characteristics and environmental conditions at the time of the crash. There is one record per crash. Extends our data here byeond NYC to highways. interesting to test correlations with weather.
Average monthly Retail Price of Electricity by state (links to EIA's API) http://www.eia.gov/beta/api/qb.cfm?category=1012 This TS would be interesting especially since its correlates with weather, storms, and impact in cities during storms. Statewide so makes it pretty easy to add, only varies in time.
If you want, I can find more or different ones, LMK.
On Thu, Sep 18, 2014 at 2:17 PM, Jonathan Giuffrida < notifications@github.com> wrote:
Some good Illinois datasets are available in the google doc https://docs.google.com/spreadsheet/ccc?key=0Au-2OHnpwhGTdGJzUWJ2SERwVXZLeDU4Y3laWFJvNEE#gid=0. More on their way, but we can definitely start with building permits and graffiti.
Note: all of these seem to stop in Jan 2013. Anyone know why?
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56088215.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389 — Reply to this email directly or view it on GitHub.
brettjgoldstein
commented
10 years ago red wine
Brett Goldstein Senior Fellow in Urban Science Harris School of Public Policy University of Chicago @bjgol
On September 18, 2014 at 3:13:46 PM, cecat (notifications@github.commailto:notifications@github.com) wrote:
Everyone....
Just an advance THANK YOU for the hard work you are doing to prep for what should be an exciting launch with Brett's talk next week. I am in DC today in in two separate gatherings the concept of Plenar.io was really excitedly received. One was a group of folks from OSTP, State Dept, a few companies, MacArthur Foundation, NIST, etc. talking about instrumenting cities to understand climate change. The other was a room full of 200 people from academic supercomputer centers, but more importantly a bunch of folks from NSF who were delighted to hear that a couple of relatively modest awards could have such impact.
I propose that after Brett returns from SFO we hold a pizza/beer celebration of year 1 of this 2 year grant.
CeC
On Sep 18, 2014, at 4:06 PM, meemking notifications@github.com wrote:
I know we want to be realistic in how many datasets are added before launch, so here are two interesting federal data sets that dovetail off of the queries and topics we have covered so far but add breadth to Plenario in that they cover stuff (1) outside of Chicago and (2) non-Socrata based and (3) specifically sourced from federal databases.
Federal Car Accident Data http://catalog.data.gov/dataset/accidents-fars-national - (1975-current): contains information about crash characteristics and environmental conditions at the time of the crash. There is one record per crash. Extends our data here byeond NYC to highways. interesting to test correlations with weather.
Average monthly Retail Price of Electricity by state (links to EIA's API) http://www.eia.gov/beta/api/qb.cfm?category=1012 This TS would be interesting especially since its correlates with weather, storms, and impact in cities during storms. Statewide so makes it pretty easy to add, only varies in time.
If you want, I can find more or different ones, LMK.
On Thu, Sep 18, 2014 at 2:17 PM, Jonathan Giuffrida < notifications@github.com> wrote:
Some good Illinois datasets are available in the google doc https://docs.google.com/spreadsheet/ccc?key=0Au-2OHnpwhGTdGJzUWJ2SERwVXZLeDU4Y3laWFJvNEE#gid=0. More on their way, but we can definitely start with building permits and graffiti.
Note: all of these seem to stop in Jan 2013. Anyone know why?
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56088215.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389 — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56095354.
cecat
commented
10 years ago Even better
CeC
On Sep 18, 2014, at 4:23 PM, brettjgoldstein notifications@github.com wrote:
red wine
Brett Goldstein Senior Fellow in Urban Science Harris School of Public Policy University of Chicago @bjgol
On September 18, 2014 at 3:13:46 PM, cecat (notifications@github.commailto:notifications@github.com) wrote:
Everyone....
Just an advance THANK YOU for the hard work you are doing to prep for what should be an exciting launch with Brett's talk next week. I am in DC today in in two separate gatherings the concept of Plenar.io was really excitedly received. One was a group of folks from OSTP, State Dept, a few companies, MacArthur Foundation, NIST, etc. talking about instrumenting cities to understand climate change. The other was a room full of 200 people from academic supercomputer centers, but more importantly a bunch of folks from NSF who were delighted to hear that a couple of relatively modest awards could have such impact.
I propose that after Brett returns from SFO we hold a pizza/beer celebration of year 1 of this 2 year grant.
CeC
On Sep 18, 2014, at 4:06 PM, meemking notifications@github.com wrote:
I know we want to be realistic in how many datasets are added before launch, so here are two interesting federal data sets that dovetail off of the queries and topics we have covered so far but add breadth to Plenario in that they cover stuff (1) outside of Chicago and (2) non-Socrata based and (3) specifically sourced from federal databases. \
Federal Car Accident Data http://catalog.data.gov/dataset/accidents-fars-national - (1975-current): contains information about crash characteristics and environmental conditions at the time of the crash. There is one record per crash. Extends our data here byeond NYC to highways. interesting to test correlations with weather.
Average monthly Retail Price of Electricity by state (links to EIA's API) http://www.eia.gov/beta/api/qb.cfm?category=1012 This TS would be interesting especially since its correlates with weather, storms, and impact in cities during storms. Statewide so makes it pretty easy to add, only varies in time.
If you want, I can find more or different ones, LMK.
On Thu, Sep 18, 2014 at 2:17 PM, Jonathan Giuffrida < notifications@github.com> wrote:
Some good Illinois datasets are available in the google doc https://docs.google.com/spreadsheet/ccc?key=0Au-2OHnpwhGTdGJzUWJ2SERwVXZLeDU4Y3laWFJvNEE#gid=0. More on their way, but we can definitely start with building permits and graffiti.
Note: all of these seem to stop in Jan 2013. Anyone know why?
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56088215.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389 — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56095354. — Reply to this email directly or view it on GitHub.
derekeder
commented
10 years ago @cecat sounds great!
@lucaluca We're adding Graffiti and Demolition Permits from IL. We'll see what else we can get to today, but I'd like to be done adding datasets by the end of the day so we have time to test performance without hitting the database with INSERTS.
meemking
commented
10 years ago Is anything CKAN in right now?
On Thu, Sep 18, 2014 at 3:59 PM, Derek Eder notifications@github.com wrote:
@cecat https://github.com/cecat sounds great!
@lucaluca https://github.com/lucaluca We're adding Graffiti and Demolition Permits from IL. We'll see what else we can get to today, but I'd like to be done adding datasets by the end of the day so we have time to test performance without hitting the database with INSERTS.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56102293.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389
derekeder
commented
10 years ago @meemking no CKAN. The datasets you linked to from Data.gov are APIs, not CSV
jcgiuffrida
commented
10 years ago Denver crimes are CKAN.
evz
commented
10 years ago @lucaluca CSV link seems broken: http://data.denvergov.org/dataset/city-and-county-of-denver-crime. Well, it only gets a header row
jcgiuffrida
commented
10 years ago Sorry about that: http://data.denvergov.org/download/gis/crime/csv/crime.csv
I could swear I saw that dataset in Plenario earlier today.
I think that's the only CKAN document we have right now. Let's try to find one or two more, if possible.
meemking
commented
10 years ago Right, they don't provide CSVs for direct download for a lot of federal data. Is the issue w/ creating a CSV from this data just a time constraint?
On Thu, Sep 18, 2014 at 4:21 PM, Jonathan Giuffrida < notifications@github.com> wrote:
Sorry about that: http://data.denvergov.org/download/gis/crime/csv/crime.csv
I could swear I saw that dataset in Plenario earlier today.
I think that's the only CKAN document we have right now. Let's try to find one or two more, if possible.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56105104.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389
evz
commented
10 years ago It's mainly a sustainability constraint. We have a thing that ingests arbitrary CSV. One could certainly take these endpoints and convert them into CSV but when it comes to regular updates, there's nothing in place that would perform that conversion on the fly.
meemking
commented
10 years ago yeah, that makes sense. unfortunately, that leaves us with limited options among federal data. Here's a link http://catalog.data.gov/dataset?metadata_type=geospatial&ext_prev_extent=-139.21874999999997%2C8.754794702435605%2C-61.87499999999999%2C61.77312286453148&res_format=CSV&q=&ext_location=&organization_type=Federal+Government&sort=views_recent+desc&ext_bbox= to the 16 geospatial federal datasets that sit in a csv format at the ready. about half are the weather data I think we already have. @lucaluca do you think any of these others are worth adding to meet the CKAN /federal data goal?
On Thu, Sep 18, 2014 at 4:40 PM, Eric van Zanten notifications@github.com wrote:
It's mainly a sustainability constraint. We have a thing that ingests arbitrary CSV. One could certainly take these endpoints and convert them into CSV but when it comes to regular updates, there's nothing in place that would perform that conversion on the fly.
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56107383.
Maggie King email: Maggiek@uchicago.edu phone: (574) 286-9389
jcgiuffrida
commented
10 years ago Here's another CKAN dataset candidate - which would get pulled into the 311 example query:
derekeder
commented
10 years ago In order to focus on performance and load testing, we have stopped loading in datasets. This means we we won't get the Denver 311 in data for the demo. We have to close this issue and prepare with what we have already.
derekeder
commented
10 years ago Here's the final list of 42 datasets. Please note that we don't have weather observations for everything, but will get it in for the NEMO and NYC motor crash demo queries
| Dataset | Source | Weather? |
|---|---|---|
| Environmental Complaints | Chicago Department of Public Health | yes |
| Environmental Hold on City-Issued Permits and LUST NFR | Chicago Department of Public Health | yes |
| Environmental Inspections | Chicago Department of Public Health | |
| Crimes - 2001 to present | Chicago Police Department | |
| Ridership - Avg. Weekday Bus Stop Boardings in October 2012 | Chicago Transit Authority (CTA) | |
| Red Light Camera Tickets | Chicago Tribune | |
| 311 Unified Data | City of Austin | |
| Winter Storm Nemo Snow Related Calls: 2/8/2013-2/10/2013 | City of Boston | pending |
| 311 Service Requests - Abandoned Vehicles | City of Chicago | |
| 311 Service Requests - Alley Lights Out | City of Chicago | yes |
| 311 Service Requests - Garbage Carts | City of Chicago | yes |
| 311 Service Requests - Graffiti Removal | City of Chicago | yes |
| 311 Service Requests - Pot Holes Reported | City of Chicago | yes |
| 311 Service Requests - Rodent Baiting | City of Chicago | yes |
| 311 Service Requests - Sanitation Code Complaints | City of Chicago | |
| 311 Service Requests - Street Lights - All Out | City of Chicago | yes |
| 311 Service Requests - Street Lights - One Out | City of Chicago | yes |
| 311 Service Requests - Tree Debris | City of Chicago | |
| 311 Service Requests - Tree Trims | City of Chicago | yes |
| 311 Service Requests - Vacant and Abandoned Buildings Reported | City of Chicago | yes |
| Building Permits | City of Chicago | yes |
| Business Licenses | City of Chicago | yes |
| Food Inspections | City of Chicago | yes |
| Micro-Market Recovery Program - Cases | City of Chicago | yes |
| Micro-Market Recovery Program - Permits | City of Chicago | yes |
| Micro-Market Recovery Program - Violations and Inspections | City of Chicago | |
| 311 Service Requests from 2010 to Present | City of New York | |
| Building Permits Since 2009 | City of Rockford | |
| Graffiti | City of Rockford | |
| Asbestos Abatement Permits - 2012 Q3 | Cook County Department of Environmental Control | |
| Asbestos Demolition Permits - 2012 Q3 | Cook County Department of Environmental Control | |
| Foreclosures - 2011 | Cook County Recorder of Deeds | |
| Foreclosures - 2012 January 1 to August 9 | Cook County Recorder of Deeds | |
| Mortgages - 2011 | Cook County Recorder of Deeds | |
| Mortgages - 2012 January 1 to August 9 | Cook County Recorder of Deeds | |
| Public Works Prohibited Contractors | Illinois Department of Labor | |
| NYPD Motor Vehicle Collisions | New York Police Department (NYPD) | pending |
| Case Data from San Francisco 311 (SF311) | San Francisco 311 | |
| Graffiti - 30 Days | San Francisco 311 | |
| Mobile Food Schedule | San Francisco Department of Public Works | |
| San Francisco Police - Crime Incidents | San Francisco Police Department (SFPD) | |
| Disabled Parking | SFMTA, SFpark |
jcgiuffrida
commented
10 years ago Are we appending weather to everything this weekend or just keeping it like this? If the latter, could we add Chicago crimes? Weather and crime is a recurring topic for us (I was thinking of putting one of those queries on the forthcoming /examples page).
On Sep 19, 2014, at 3:18 PM, Derek Eder notifications@github.com wrote:
Here's the final list of datasets. Please note that we don't have weather observations for everything, but will get it in for the NEMO and NYC motor crash demo queries Dataset Source Weather? Environmental Complaints Chicago Department of Public Health yes Environmental Hold on City-Issued Permits and LUST NFR Chicago Department of Public Health yes Environmental Inspections Chicago Department of Public Health Crimes - 2001 to present Chicago Police Department Ridership - Avg. Weekday Bus Stop Boardings in October 2012 Chicago Transit Authority (CTA) Red Light Camera Tickets Chicago Tribune 311 Unified Data City of Austin Winter Storm Nemo Snow Related Calls: 2/8/2013-2/10/2013 City of Boston pending 311 Service Requests - Abandoned Vehicles City of Chicago 311 Service Requests - Alley Lights Out City of Chicago yes 311 Service Requests - Garbage Carts City of Chicago yes 311 Service Requests - Graffiti Removal City of Chicago yes 311 Service Requests - Pot Holes Reported City of Chicago yes 311 Service Requests - Rodent Baiting City of Chicago yes 311 Service Requests - Sanitation Code Complaints City of Chicago 311 Service Requests - Street Lights - All Out City of Chicago yes 311 Service Requests - Street Lights - One Out City of Chicago yes 311 Service Requests - Tree Debris City of Chicago 311 Service Requests - Tree Trims City of Chicago yes 311 Service Requests - Vacant and Abandoned Buildings Reported City of Chicago yes Building Permits City of Chicago yes Business Licenses City of Chicago yes Food Inspections City of Chicago yes Micro-Market Recovery Program - Cases City of Chicago yes Micro-Market Recovery Program - Permits City of Chicago yes Micro-Market Recovery Program - Violations and Inspections City of Chicago 311 Service Requests from 2010 to Present City of New York Building Permits Since 2009 City of Rockford Graffiti City of Rockford Asbestos Abatement Permits - 2012 Q3 Cook County Department of Environmental Control Asbestos Demolition Permits - 2012 Q3 Cook County Department of Environmental Control Foreclosures
— Reply to this email directly or view it on GitHub https://github.com/datamade/plenario/issues/94#issuecomment-56229343.
derekeder
commented
10 years ago @lucaluca we'll look in to adding it, but no promises on new data at this point.
brettjgoldstein
commented
10 years ago important — lmk
Brett Goldstein Senior Fellow in Urban Science Harris School of Public Policy University of Chicago @bjgol
On September 19, 2014 at 3:25:58 PM, Derek Eder (notifications@github.commailto:notifications@github.com) wrote:
@lucalucahttps://github.com/lucaluca we'll look in to adding it, but no promises on new data at this point.
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56230702.
derekeder
commented
10 years ago Ok weather data is loaded in for:
brettjgoldstein
commented
10 years ago Great
On Sat, Sep 20, 2014 at 6:08 PM, Derek Eder notifications@github.com wrote:
Ok weather data is loaded in for:
— Reply to this email directly or view it on GitHubhttps://github.com/datamade/plenario/issues/94#issuecomment-56283169.
We currently have 37 datasets loaded in to Plenar.io. For the purposes of demoing to a wide audience, I think we should pare this list down to the datasets that show off Plenar.io's capabilities. Related to #93