search

koopjs / koop-pgcache

PostGIS cache for Koop.
Other
4 stars 4 forks source link

add streaming export factory method #53

Closed dmfenton closed 8 years ago

dmfenton commented 8 years ago

This PR adds new functionality to create a stream of features (as JSON or strings) that come directly from the DB. Under the hood it uses psql to execute a copy command and Highland.js to prepare each feature for consumption.

var options = { where: 'id > 10', json: true} // options include filter clauses and json vs string emittance
var stream = pgCache.createExportStream('tableName', options)
jakesower commented 8 years ago

So a different implementation of getting data into the database? It looks to me like that process is one that silently hums along unless an error occurs. Is that a correct assessment?

dmfenton commented 8 years ago

It's actually for getting data out. It will emit features one by one in a stream unless an error occurs.

jakesower commented 8 years ago

Ah, I see. This isn't the ultimate consumer of the data, it just throws JSON out to somewhere else. I'll have to read up on highland more carefully at some point.

dmfenton commented 8 years ago

Yep. JSON or strings, depending on how you want to consume the data.

The first use case for me is creating a geojson feature collection -> gzipping -> uploading to S3 all in a stream.