search

codeforokc / school-finder

Geolocation-based web app for locating schools and school districts near you
MIT License
18 stars 17 forks source link

Add initial data ingest #7

Closed joekarl closed 9 years ago

joekarl commented 9 years ago

Refactored @adamveld12's code a bit and have it outputting geojson. There's no upload capabilities or anything like that, but just a start. Somewhat setup to add more data parsers but could certainly be cleaned up better for that.

Also added the derived sample data in the data folder.

joekarl commented 9 years ago

//cc @jvrousseau

joekarl commented 9 years ago

Well I didn't convert the meters to lat lon so the geojson is hosed, so extra work needed on this :/

joekarl commented 9 years ago

Much better. Should be good now. Using proj4 to do meters to lat lon and do reprojection at same time.

jvrousseau commented 9 years ago

Awesome! I'll take a quick look and we can also compare geojson output with the kmz file to ensure an accurate conversion/re-projection. Looking great @joekarl

joekarl commented 9 years ago

@jvrousseau the github rendering of geojson looks legit (https://github.com/codeforokc/school-finder/blob/data_ingest/data_ingest_node/data/schoolDistricts.geojson), but yeah some spot checking would be good.

joekarl commented 9 years ago

The data is actually really really clean. What we'll need to do is come up with a mapping of school to school district (that info isn't in the data we have from the city :/ )

DevinClark commented 9 years ago

@joekarl that may be something Turf could handle. http://rousseau.io/turf-mapboxjs/turf-inside/

joekarl commented 9 years ago

Uggh.. uncompressed that data is like 400kb+ and 750kb+ respectively. That sucks. Luckily thats uncompressed prettified json. Gzipping should be able to get it to around 100kb or so, but that's still pretty bad for mobile... Probably can cut a lot of data if we remove all of the properties from school/school district features. We can store them compressed outside of the geojson (we'll probably need to do some of that anyways to map schools to school district). May also have to truncate precision on lat lon coordinates as well.

Really would like to get it down to like 25kb or so if possible...

DevinClark commented 9 years ago

You could also do some pretty heavy caching on the files. I assume this data doesn't change very often?

joekarl commented 9 years ago

@DevinClark yeah, problem is it'll be hard to cache on the client for mobile (which is a major target). Hence the need to get the data as small as possible...

jvrousseau commented 9 years ago

@joekarl Do you think it would be better to try to find or compute schools as points rather than polygons of their campus? I think the district dataset could be simplified and precision knocked down a bit without losing anything. But I'm not sure if we have to have schools as polygons. What are your thoughts?

joekarl commented 9 years ago

@jvrousseau yeah both would significantly help. I don't think we need schools as polygons (we'll probably throw that data away anyways in the UI). The size of a geojson point is significantly smaller than a geojson polygon (or multipolygon).

jagthedrummer commented 9 years ago

+1 for reducing schools to a point.

-1 for truncating precision on district boundaries. If we did that I think we'd have a lot of problems in finding the right district when we're close to an edge.