search

dat-ecosystem-archive / datproject-discussions

a repo for discussions and other non-code organizing stuff [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
65 stars 6 forks source link

geopipes #47

Open joehand opened 8 years ago

joehand commented 8 years ago

From @missinglink on July 21, 2014 21:29

Hey @maxogden, team.

As we discussed the other day at nodeconf in London, I'm currently leading a project at mapzen to create a set of curated stream libraries for working specifically with geographic datasets.

My personal interest in creating the libraries is to facilitate importing of large amounts of POIs in to elasticsearch for our opensource/opendata geosearch engine called Pelias.

However, outside that; it'd be great to have a set of stream libraries for dealing with these large geo datasets which can empower other developers to build their own geo pipelines.

I've just created the geopipes organisation on github today and I added an elasticsearch backend that should be compatible with dat, if you're interested in that sort of thing. https://github.com/geopipes/elasticsearch-backend

disclaimer: I only wrote that this morning, it has pretty full test coverage but I haven't given it a thorough beating yet.

Over the next couple of weeks I will add the repos for openstreetmap, geonames and quattroshapes as well as streaming parsers for osm pbf and shapefiles. (mostly written by others)

max asked me to open a ticket to discuss if we can find some way of working together and to make you aware of what we're working on.

Copied from original issue: maxogden/dat#140

joehand commented 8 years ago

From @missinglink on July 21, 2014 21:34

This is the issue I mentioned about substacks osm-pbf-parser library which is preventing me from adopting it. https://github.com/substack/osm-pbf-parser/issues/5

at the moment, I am using this instead: https://github.com/marook/osm-read/issues/22

joehand commented 8 years ago

From @jden on July 21, 2014 22:3

@missinglink sounds super cool! Is there a good primer or anything written which explains the geopipes idea in more detail?

joehand commented 8 years ago

From @missinglink on July 23, 2014 16:58

Hey @jden, sorry I haven't got a primer written yet. BUT!, I just shipped a module for working with geonames: https://github.com/geopipes/geonames-stream

Is this sort of stuff compatible with dat?

joehand commented 8 years ago

From @missinglink on July 23, 2014 18:42

osm: https://github.com/geopipes/openstreetmap-stream

joehand commented 8 years ago

From @jden on July 23, 2014 18:45

Okay - I think I understand more now. Geopipes is more of a set of tools, rather than a format or a protocol, is this correct?

joehand commented 8 years ago

From @missinglink on July 23, 2014 18:47

yep. it's simply a project to create a set of streaming libraries for geo related datasets

joehand commented 8 years ago

From @maxogden on July 24, 2014 13:13

I just wrote up an initial draft of a guide on importing data into dat https://github.com/maxogden/dat/blob/master/docs/importing.md

I think geopipes is definitely gonna be useful

joehand commented 8 years ago

From @maxogden on July 24, 2014 13:13

Also: https://github.com/datproject/meta/issues/5

joehand commented 8 years ago

From @ryanramage on December 2, 2014 16:52

Wrote this recently, https://www.npmjs.org/package/reproject-stream noting here as there is no 'official list' of these types of tools. Maybe someone will find useful.