Closed maurizi closed 2 years ago
Well this was an all-around disappointment.
Testing in staging showed using topobuf instead of v8 deserialization consumes more memory: (left side is using v8, right is using pbf)
This is probably something we could fix in the topobuf library at the expense of some CPU cycles... but we're slower than v8 deserialization already 😞
topobuf:
districtbuilder-server-1 | cache-miss San Bernadino: 227.267ms
districtbuilder-server-1 | cache-miss Dane County: 634.601ms
districtbuilder-server-1 | cache-miss Delaware: 826.569ms
districtbuilder-server-1 | cache-miss New Jersey: 3.451s
districtbuilder-server-1 | cache-miss Wisconsin: 5.165s
districtbuilder-server-1 | cache-miss Pennsylvania: 14.393s
districtbuilder-server-1 | cache-miss Illinois: 10.048s
v8:
districtbuilder-server-1 | cache-miss San Bernadino: 137.446ms
districtbuilder-server-1 | cache-miss Dane County: 327.031ms
districtbuilder-server-1 | cache-miss Delaware: 597.973ms
districtbuilder-server-1 | cache-miss New Jersey: 3.232s
districtbuilder-server-1 | cache-miss Wisconsin: 5.475s
districtbuilder-server-1 | cache-miss Pennsylvania: 10.225s
districtbuilder-server-1 | cache-miss Illinois: 8.007s
File size is much much smaller with topobuf, but that is about the only thing it has going for us. Probably worth looking into topobuf further for potentially sending TopoJSON to the browser due to the much smaller file size, but ultimately I think we should stick with the current serialization method.
Context
Work on the concurrency ADR showed that we can get significant improvements in file size by using a Protobuffer based file format - I used an outdated version of
geobuf
to do this - we'll be forking that in #1141 astopobuf
Notes
We changed from JSON -> the
v8
serialization format in https://github.com/PublicMapping/districtbuilder/pull/1099, which should be able to serve as a template for this PRAC:
process-geojson
and the backend server code to usetopobuf
instead ofv8.serialize
/v8.deserialize
serialize-topojson
to be able to readv8
files and writetopobuf
filesserialize-topojson
script to make new.pbf
files for every region in:scripts/load-dev-data