No "transform" values after converting geoJson to topoJson with geo2topo

[normonics]

Hi,

I was using the old topojson command line tool, which always worked well. Now, using geo2topo I'm having a number of issues.

One of the issues is that the topojson output seems to lack the "transform" dictionary containing "scale" and "translate"

Best, Joe

[mbostock]

Please post code to reproduce the issue if you’d like me to investigate. Thank you!

[normonics]

I was following the 'lets make a map tutorial' previously to get from shapefile to topojson

my topojson files had in the "geometries" object a "transform" object, e.g.:

"transform": { "scale": [ 0.0000035892581358925812, 8.59051108590511E-7 ], "translate": [ -179.14734, -14.552549 ]

now when using geo2topo it gives me a 'bbox' instead, and my existing javascript doesn't know what to do with it anymore.

"bbox": [ -179.14733999999999, -14.552548999999999, 179.77847, 71.352561 ]

does that make sense? is that what should be happening?

Thanks for your help, Mike.

-Joe

[normonics]

This trouble all began when i was unable to convert a ~350MB GeoJSON file to TopoJSON using the old topojson command line tool. So I updated via NPM. Now everything seems out of whack.

I'm a noob when it comes to managing all these NPM packaged and dependencies etc. ANY help getting my stuff together would be greatly appreciated. is the large filesize a known issue> I know you're not updating topojson command line but rather geo2topo but obviously I'm having troulb with thtat, too...

(I should add I have same file-size problem when using geo2topo. I tried trimming the geo file down significantly by only leaving 1000 features in, which allowed a conversion but ended up without the "transform" object which is where this thread started. sorry for the longwindedness)

[normonics]

error when trying to use geo2topo on ~350MB GeoJson

$ geo2topo parcels.json > parcels_topo.json
buffer.js:609
    return this.utf8Slice(0, this.length);
                ^

Error: "toString()" failed
    at Buffer.toString (buffer.js:609:17)
    at JSON.parse (<anonymous>)
    at ReadStream.<anonymous> (/usr/local/lib/node_modules/topojson/node_modules/topojson-server/bin/geo2topo:107:46)
    at emitNone (events.js:111:20)
    at ReadStream.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1056:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[normonics]

Will be my last clarifying post hoping for your help here, sorry for the bombardment

when I avoid the issue of file size, I end up with a topojson file that has NO transform object.

e.g., a file I generated previously has the following keys:

[u'objects', u'type', u'transform', u'arcs']

file I generate with geo2topo has the following keys:

[u'objects', u'type', u'bbox', u'arcs']

[mbostock]

The lack of a transform is expected because geo2topo doesn’t perform quantization by default; a TopoJSON topology only has a transform property if it is quantized; see section 2.1.2 of the specification.

To create a quantized topology, you typically specify the -q or --quantization argument to geo2topo. This quantizes the input geometry before the topology is constructed, which is nice if you have input that isn’t exactly topologically consistent. (This is equivalent to the old )

If you want to quantize the topology after construction (to quantize without affecting the topology), you should instead pipe the output of geo2topo to topoquantize.

[mbostock]

The error calling JSON.parse is because the file is too big to fit in memory. I believe you need approximately 3x the amount of memory as the input file size, because Buffer.concat creates a copy, followed by JSON.parse.

https://github.com/topojson/topojson-server/blob/c33f1f0d66ff43fcc3bf854aafce9c6fd60945d3/bin/geo2topo#L102-L110

And by default (according to Stack Overflow), Node appears to have either a 512MB or 1GB memory limit. So it’s quite plausible that you are hitting this limit.

There are two ways you can workaround this issue. These are independent solutions; you can apply both of them simultaneously if you like.

The first is to increase the amount of memory available to Node. For example, you might run geo2topo as follows:

node --max-old-space-size=8192 `which geo2topo` parcels.json > parcels_topo.json

The second approach is use newline-delimited JSON as input, so that you can parse each feature of your input GeoJSON separately, rather than needing to load and parse the entire file at once. That requires specify the -n or --newline-delimited flag.

geo2topo -n parcels.ndjson > parcels_topo.json

This, of course, requires that you convert your GeoJSON to newline-delimited GeoJSON first. For smaller files, the easiest way to do that is ndjson-cat and ndjson-split from the ndjson-cli. Or, if you have a shapefile, use shp2json -n.

A thing we could do to help internally is to switch to JSONStream to parse the GeoJSON input more efficiently. I’m going to look into that now and I will file a pull request if I make progress.

[mbostock]

I spent a little time looking at clarinet, bfj and jsonparse, and unfortunately none of them seem to work very well for our needs. So I think your best option is probably newline-delimited JSON.