search

mapbox / shp-write

create and write to shapefiles in pure javascript
BSD 3-Clause "New" or "Revised" License
292 stars 186 forks source link

Streaming output #11

Closed tmcw closed 10 years ago

tmcw commented 10 years ago

Is it possible to stream typed arrays? Unclear!

newmanw commented 10 years ago

Any answer? I have a huge collection of geojson points I would like to stream to shapfiles. I am essentially paging through my results to save on memory and would like to write the results to stream as I get them.

tmcw commented 10 years ago

"Is it possible to stream typed arrays?": No. Typed arrays are allocated, and streaming them is a big hack that's not very worthwhile.

@newmanw unless you need it to be a shapefile for some reason, it would be better to stream your features into a spatial database like SQLite or PostGIS rather than using shapefiles at all.

newmanw commented 10 years ago

@tmcw unfortunately my store is mongo and I have people looking to export my data to ESRI and the preferred format for them is shapefiles :(

For now I am just creating a shapefile for each 'page', a bit of a hack but not much else I can do.

tmcw commented 10 years ago

You can stream into a geojson featurecollection, or use esri's sqlite support?

newmanw commented 10 years ago

Does ESRI read/parse GeoJSON? Cause yes of course a feature collection would easy, just would take a little convincing to have customer pull in GeoJSON over ESRI. Always a battle to get folks to move away from ESRI shapefiles.

newmanw commented 10 years ago

I do have a private implementation that I was messing around with before I saw this issue. I have modified this library to write to a file stream and it works. Biggest issue I am running into with streaming is the ability to dynamically keep updating the extent, and length as I write to the stream.

I am starting to wonder if we are talking about the same thing, or if I have just confused the issue ;)

Given a set of 2000 (one page) results I am currently using a modified version of this to stream to a file rather than write it all in memory. Works just great an wondering with some mods if I could delay the writing of some of the metadata about the collection (extent, length, etc) until I know I am done paging.