Add transmission lines overlay

[cypressf]

Alyssa says:

Hi Cypress, I uploaded two zipped files to the Drop Box folder: Voltage230_345 - shape file of voltage >=230kV and <345kV (Label this 'Second level of transmission 230kV-345kV' and Color this on GREEN) Voltage_over_345 - shape file of voltage >=345kV (Label this on 'Third level of transmission >= 345kV and Color this on RED)

The link to this data set and description is the in the file 'Notes on Indicators'

If you have any questions please let me know. If these files are too big and I need to try to cut out unnecessary columns, let me know and I can see if that helps.

[cypressf]

I tried using shp2json Voltage230_345/Voltage230_345.shp > geojson.json and the resulting geojson file was 16 MB, which resulted in an svg too-large to render in chrome.

[cypressf]

hmm... upon closer inspection, that lack of displaying the geojson may have been due to the bounding box not being set correctly. I tried displaying it with mapshaper and I got the following

[cypressf]

I think projecting it using d3.geoAlbersUsa().scale(1300).translate([487.5, 305]) breaks things. I've tried doing that, and the file size inflates from 16MB to 42MB, and the lines are all over the place (basically fills the viewport black)

[cypressf]

Yes, similarly I can get it down to 2.9MB by

shp2json $argv[1] |
    geoproject 'd3.geoAlbersUsa().scale(1300).translate([487.5, 305])' |
    ndjson-split 'd.features' |
    ndjson-map 'delete d.properties.OBJECTID,
delete d.properties.ID,
delete d.properties.TYPE,
delete d.properties.STATUS,
delete d.properties.NAICS_CODE,
delete d.properties.NAICS_DESC,
delete d.properties.SOURCE,
delete d.properties.SOURCEDATE,
delete d.properties.VAL_METHOD,
delete d.properties.VAL_DATE,
delete d.properties.OWNER,
delete d.properties.VOLTAGE,
delete d.properties.VOLT_CLASS,
delete d.properties.INFERRED,
delete d.properties.SUB_1,
delete d.properties.SUB_2,
delete d.properties.SHAPE_Leng,
d' >temp-map.ndjson
geo2topo -n electric_lines=temp-map.ndjson |
    toposimplify -p -1 -f |
    topoquantize 1e5 >electric-lines-topo.json

This works as long as I don't try to project using geoproject 'd3.geoAlbersUsa().scale(1300).translate([487.5, 305])'

[cypressf]

I tried projecting the dataset from HIFLD https://hifld-geoplatform.opendata.arcgis.com/datasets/electric-power-transmission-lines and that worked with the projection. I will go ahead and use that dataset for now. It's still too big but I'll look into ways to make it smaller.

[cypressf]

I added the electric line data and transformation and projection script to https://github.com/cypressf/climate-map-tests/commit/9a6399054b13cbe14ee23aa034dc33b3f0373ff8

[cypressf]

Okay, I added the transmission lines to the beta website. They are still very large. 2.5MB and 4.8MB. They take a long time to load, and slow the website down when they are displayed in the map. I think it would be better to find a smaller dataset, or find a way to make this dataset even smaller before publishing.

[cypressf]

To simplify geometry we would want to use something like the Ramer Douglas Peucker algorithm

[cypressf]

It looks like ArcMap can run multiple polygon-simplification algorithms (including Ramer Douglas Peucker). I'm not sure if Alyssa has access to this software. I will check.

[azzacasch]

Hey Cypress,

This looks very promising! - I’d say the “simplest” geometry we could bring it down to would be so that at the very least a user is able to discern where/which transmission lines cut cross which counties.

—a

On Jun 25, 2021, at 12:08 PM, Cypress Frankenfeld @.***> wrote:

To simplify geometry we would want to use something like the Ramer Douglas Peucker algorithm https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm https://user-images.githubusercontent.com/443397/123453989-10558d00-d5ae-11eb-8b39-1d283fce5626.gif — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cypressf/mit-climate-data-viz/issues/182#issuecomment-868672135, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYYCPKZP6ZMGORYJ3HQDE3TUSSY7ANCNFSM46MGMERQ.

[cypressf]

In addition to simplification (reducing the number of vertices in a polygon), I will use quantization to reduce the precision of each vertex (e.g., change (0.12331, 20.13219) -> (0, 20)). This is something I already do with all our topojson files, and it typically reduces their size 5-fold.

More info about simplification and quantization here: http://zevross.com/blog/2014/04/22/spatial-data-on-a-diet-tips-for-file-size-reduction-using-topojson/

[cypressf]

Looks like the topojson command line utility that I have already been using for quantization supports simplification using the Visvalingam algorithm, which they say "may be more effective" than the Douglas Peucker one. I'm going to try using their simplification algorithm which should obviate using ArcMap.

[cypressf]

Breakthrough! I was previously using toposimplify but feeding it a -p -1 instead of the (correctly formatted) -p 1, which was causing the visvalignam algorithm to not run. When I pass the correct -p 1 (simplify to the nearest pixel) flag, it reduces the svg size from 7 MB to 324 KB.

[cypressf]

example of simplifying, quantizing, and saving as svg

 geoproject 'd3.geoAlbersUsa().scale(1300).translate([487.5, 305])' < input-map.json |
                             ndjson-split 'd.features' |
                             geo2topo -n |
                             toposimplify -p 1 -f |
                             topoquantize 1e8 |
                             topo2geo - -n |
                             geo2svg -w 960 -h 960 -n > temp.svg

[cypressf]

no simplification or quantization 12.5 MB

toposimplify -p 1 -f | topoquantize 1e8 330 KB

toposimplify -p 100 -f | topoquantize 1e8 246 KB (diminishing returns with p >> 1)

toposimplify -p 1 -f | topoquantize 1e3 215 KB (diminishing returns with quantize << 1e8)

[cypressf]

I used the following command to generate the png preview, and output the topojson files so I could view the file sizes (all the delete properties lines are just to remove some misc non-geometric properties that were in the shape file)

 geoproject 'd3.geoAlbersUsa().scale(1300).translate([487.5, 305])' < FIFLD\ over\ 345.json |
                             ndjson-split 'd.features' |
                             ndjson-map 'delete d.properties.OBJECTID,
                             delete d.properties.ID,
                             delete d.properties.TYPE,
                             delete d.properties.STATUS,
                             delete d.properties.NAICS_CODE,
                             delete d.properties.NAICS_DESC,
                             delete d.properties.SOURCE,
                             delete d.properties.SOURCEDATE,
                             delete d.properties.VAL_METHOD,
                             delete d.properties.VAL_DATE,
                             delete d.properties.OWNER,
                             delete d.properties.VOLTAGE,
                             delete d.properties.VOLT_CLASS,
                             delete d.properties.INFERRED,
                             delete d.properties.SUB_1,
                             delete d.properties.SUB_2,
                             delete d.properties.SHAPE_Length,
                             d' |
                             geo2topo -n |
                             toposimplify -p 1 -f |
                             topoquantize 1e5 |
                             tee temp-topo.json |
                             topo2geo - -n |
                             geo2svg -w 960 -h 660 -n |
                             rsvg-convert -w 960 > temp.png

[cypressf]

Added to mst.mit.edu