Calculate colour_name property for transit lines

nvkelso commented 7 years ago

Right now we pass thru the raw OpenStreetMap line colors on transit layer line features. But many of those are "dirty" colors that don't pair well with basemap colors (and we have at least 6 Mapzen basemap styles now), and are a mix of HEX (#ff0000) and named colors (red).

We'd like to have an additional set of "Crayola 64" normalized line colors (keep the color we have now, but add a curated set). Maybe this is "Crayola" or maybe it's a custom set of transit colors based on what we see in the data (e.g.: blue, blue_light, blue_dark, etc). But each stylesheet could then map those normalized colors to a version that pairs with the rest of the colors in the map, without having to curate by line.

Turns out @thisisaaronland did similar work for Cooper Hewitt a couple years ago that might be useful here:

/cc @sensescape @irees

nvkelso commented 7 years ago

Let's export a big list of all the transit route relations, their colors, system operators, and line names for @sensescape to review. I'll ask Ian Rees for the same from Transit.land.

zerebubuth commented 7 years ago

Here's a list of all the colours I can find from the bus, train, railway, tram, trolleybus, light_rail and subway route relations; route_colours.txt. Note that it's actually a CSV file, but Github doesn't allow attaching those.

Although the selection of blue, green, red and black are most popular, and account for almost 15% of all entries, the majority of entries in the list are hex colours, with a few GoldenRods thrown in for good measure.

Some interesting highlights:

Apparently something has the colour *, which must mean it sparkles at night? Thankfully, I think we have a shader for that.
One thing has a colour of fixme, which I'm unfamiliar with. I must look that up in the Pantone catalogue.
It's not clear to me whether there's a route with colour ffffff, or the author was just frustrated.
Something is apparently ex:brown, although I don't know what the new colour is.

:wink:

zerebubuth commented 7 years ago

With a bit of mangling, I was able to match 99% of the colours to either hex6, hex3 or a name from the X11 RGB palette. The remainders don't seem to be that hard to match, and mainly seem to be translations of colours. See match.txt for details.

The palettes from the Cooper Hewitt seem pretty big (around 150 colours each). Even 64 colours seems like a lot to be remapping to match a colour theme. Perhaps we could start off mapping to red, orange, yellow, green, blue, purple, grey, white and black and generating light/dark variants for the non-greyscale ones? That would give us 21 colours, but perhaps only require tweaking 9 by hand.

nvkelso commented 7 years ago

Puts on my fixme hat, lol...

nvkelso commented 7 years ago

@zerebubuth could you create a CSV with the input and the output, please?

zerebubuth commented 7 years ago

route_relations.zip is a (very) large list of all route relations from OpenStreetMap, available under the ODbL.

I've extracted the route type (bus, train, etc...), operator, network, colour and name, although many of the entries won't have values for some combinations of those properties.

zerebubuth commented 7 years ago

could you create a CSV with the input and the output, please?

The inputs were route_colours.txt and X11's rgb.txt, and the output was just match.txt. I didn't actually map the colours, as we've not decided yet what we might want to map them to. It was just an exercise to check that we could parse most of them.

nvkelso commented 7 years ago

Gotcha, makes sense.

nvkelso commented 7 years ago

Just to show off Github Markdown automatic color coding, here are any with count ≥ 20 50 :)

colour	count
`blue` `#0000ff`	1344
`green` `#00ff00`	927
`red` `#ff0000`	854
`black` `#000000`	610
`#79b51d`	533
`#FF0000`	311
`#0000FF`	275
`yellow` `#FFFF00`	272
`#CD853F`	262
`#000000`	202
`#FFFF00`	194
`#a2bd31`	184
`#ff0000`	164
`#ED1C24`	163
`#008000`	162
`f77921` `#f77921`	148
`#005266`	147
`orange` `#ffa500`	143
`#BB0022`	116
`#771133`	113
`#312783`	112
`#00FF00`	105
`#FFCC11`	95
`#EE8822`	91
`brown` `#a52a2a`	82
`#E70000`	81
`#DD0077`	80
`#991199`	75
`#0099AA`	74
`#fbbf15`	72
`#BBDD00`	69
`#229922`	67
`#FF88AA`	67
`#0044BB`	64
`#995511`	64
`#0000ff`	63
`#D7B52E`	63
`#00365f`	59
`#444411`	59
`#AACCEE`	59
`#008be0`	58
`#007DC5`	57
`#FCCD51`	57
`#808080`	56
`#95276E`	54
`#FFDD00`	54
`#EC2A2F`	53
`purple` `#800080`	49
`#C5AA77` `#c5aa77`	48
`#1199DD` `#1199dd`	47
`#8899AA` `#8899aa`	46
`#005555`	45
`#009ee0`	45
`#039fe0`	44
`#D9680C` `#d9680c`	44
`#800080`	42
`#ee1f23`	42
`grey` `#808080`	40
`#a3429b`	39
`#ED222E` `#ed222e`	39
`#EA2423` `#ea2423`	38
`#000080`	37
`#e55c00`	37
`#9C2368` `#9c2368`	36
`#F39200` `#f39200`	35
`gray` `#808080`	35
`#26CEFF` `#26ceff`	34
`#00B1EE` `#00b1ee`	33
`#7FFF00` `#7fff00`	33
`#fed452`	32
`#77CCAA` `#77ccaa`	31
`#901E78` `#901e78`	31
`#e8c31b`	31
`white` `#ffffff`	31
`#003273`	30
`#00A1DA` `#00a1da`	30
`#F7D358` `#f7d358`	30
`#ffff00`	30
`magenta` `#ff00ff`	30
`#6FF` `#66ffff`	29
`#ed1c24`	29
`#f49694`	29
`#00adef`	28
`#0e8789`	28
`#424A52` `#424a52`	28
`#990066`	28
`#ADD8E6` `#add8e6`	28
`#EC6607` `#ec6607`	28
`#008DD0` `#008dd0`	27
`#53b7e8`	27
`87CEEB` `#87ceeb`	27
`#94D24D` `#94d24d`	27
`#f68712`	27
`#ff9900`	27
`#00a650`	26
`#00ADEF` `#00adef`	26
`#f2cc05`	26
`#FFA500` `#ffa500`	26
`FFFF00` `#ffff00`	26
`#8cbc00`	25
`#958f9f`	25
`#FECC00` `#fecc00`	25
`#009cdd`	24
`#3eba7f`	24
`#43c5b5`	24
`#563F00` `#563f00`	24
`#70266f`	24
`#8b004f`	24
`#91d562`	24
`LightGreen` `#90ee90`	24
`pink` `#ffc0cb`	24
`#B291FF` `#b291ff`	23
`#F1AC2D` `#f1ac2d`	23
`#F40002` `#f40002`	23
`#ffcc00`	23
`#00C62D` `#00c62d`	22
`#0166b3`	22
`mediumseagreen` `#3cb371`	22
`violet` `#ee82ee`	22
`#0083CA` `#0083ca`	21
`#0099FF` `#0099ff`	21
`#009fe3`	21
`#1080c4`	21
`#87c440`	21
`#b1c903`	21
`#e30613`	21
`#FF00FF` `#ff00ff`	21
`#ff6600`	21

nvkelso commented 7 years ago

Fun dive into subway colors:

nvkelso commented 7 years ago

Let's give these 38 colors a try:

ignore the order, tilezen name, and mz hex – those are me trying to match everything up.
use the css name and css hex instead.
the list is the core 16 original CSS colors (which are similar to the original 16 Macintosh colors), plus a range of light and dark variants, and some odd balls added to based on the data sample.

order	tilezen name	mz `hex`	css name	css hex
00	white	`#ffffff`	`white`	`#ffffff`
01	yellow	`#ffff00`	`yellow`	`#ffff00`
24	gold	`#ffcc11`	`gold`	`#ffd700`
02	orange	`#ffa500`	`orange`	`#ffa500`
18	lightcoral	`#f08080`	`lightcoral`	`#f08080`
16	sandybrown	`#fbbf15`	`sandybrown`	`#f4a460`
17	peru	`#d9680c`	`peru`	`#cd853f`
34	crimson	`#dc143c`	`crimson`	`#dc143c`
03	red	`#dd0907`	`red`	`#ff0000`
33	tomato	`#ff6347`	`tomato`	`#ff6347`
28	hotpink	`#ff69b4`	`hotpink`	`#ff69b4`
04	fuchsia	`#f20884`	`fuchsia`	`#ff00ff`
20	violet	`#ee82ee`	`violet`	`#ee82ee`
23	plum	`#dda0dd`	`plum`	`#dda0dd`
19	pink	`#ffc0cb`	`pink`	`#ffc0cb`
33	darkorchid	`#9932cc`	`darkorchid`	`#9932cc`
05	purple	`#991199`	`purple`	`#800080`
06*	blue	`#0000d3`	`blue`	`#0000ff`
30	dodgerblue	`#1e90ff`	`dodgerblue`	`#1e90ff`
29	royalblue	`#4169e1`	`royalblue`	`#4169e1`
31	steelblue	`#4682b4`	`steelblue`	`#4682b4`
25	teal	`#008080`	`teal`	`#008080`
26	lightteal	`#0099aa`	`lightteal`	`#0099aa`
21	lightblue	`#add8e6`	`lightblue`	`#add8e6`
07	aqua	`#02abea`	`aqua`	`#00ffff`
07	aquamarine	`#7fffd4`	`aquamarine`	`#7fffd4`
22	darkseagreen	`#a2bd31`	`darkseagreen`	`#8fbc8f`
32	olivedrab	`#6b8e23`	`olivedrab`	`#6b8e23`
35	yellowgreen	`#9acd32`	`yellowgreen`	`#9acd32`
08*	limegreen	`#79b51d`	`limegreen`	`#32cd32`
09	green	`#008000`	`green`	`#008000`
10	tan	`#cd853f`	`tan`	`#d2b48c`
11	brown	`#90713a`	`brown`	`#a52a2a`
27*	gainsboro	`#dcdcdc`	`gainsboro`	`#dcdcdc`
12	silver	`#c0c0c0`	`silver`	`#c0c0c0`
13	grey	`#808080`	`grey`	`#808080`
14	darkgrey	`#404040`	`darkgrey`	`#404040`
15	black	`#000000`	`black`	`#000000`

Notes / also-known-as:

blue above could also be described as mediumblue
sandybrown could be also be described as lightorange
gainsboro is a warm grey
aqua is also cyan
fushia is also magenta

nvkelso commented 7 years ago

Also note that this code will initially just target the transit layer, which means bus and trolley_bus will generally be excluded. (And bus lines should generally should be symbolized with level of service / frequency not color.)

nvkelso commented 7 years ago

For relations that don't have a color, let's specify a a named "default" color per transit mode:

default_train
default_railway
default_tram
default_light_rail
default_subway

Note: bus and trolleybus aren't included in the transit layer so are excluded here.

nvkelso commented 7 years ago

I think we strip of mz_ prefixed values so what should we call this new, additional colour property?

zerebubuth commented 7 years ago

Here's what applying the colour palettes looks like to the top 20 colours:

name	count	original colour	NVK color	MLA colour
blue	1344	`#8a2be2`	`#9932cc`	`#a500ff`
green	927	`#adff2f`	`#9acd32`	`#80ff80`
red	854	`#ff0000`	`#dd0907`	`#ff0000`
black	610	`#000000`	`#000000`	`#000000`
#79b51d	533	`#79b51d`	`#79b51d`	`#808000`
#FF0000	311	`#ff0000`	`#dd0907`	`#ff0000`
#0000FF	275	`#0000ff`	`#0000d3`	`#0000ff`
yellow	272	`#ffff00`	`#ffff00`	`#ffff00`
#CD853F	262	`#cd853f`	`#cd853f`	`#ff8080`
#000000	202	`#000000`	`#000000`	`#000000`
#FFFF00	194	`#ffff00`	`#ffff00`	`#ffff00`
#a2bd31	184	`#a2bd31`	`#a2bd31`	`#808000`
#ff0000	164	`#ff0000`	`#dd0907`	`#ff0000`
#ED1C24	163	`#ed1c24`	`#dc143c`	`#ff0000`
`#008000`	162	`#008000`	`#008000`	`#008000`
f77921	148	`#f77921`	`#d9680c`	`#ffa500`
#005266	147	`#005266`	`#008080`	`#000080`
orange	143	`#ff4500`	`#d9680c`	`#ff0000`
#BB0022	116	`#bb0022`	`#dd0907`	`#800000`
#771133	113	`#771133`	`#404040`	`#800000`

The full list is available as palette.txt.

nvkelso commented 7 years ago

I'm confused how:

blue with 1344 counts converted to #8a2be2 instead of #0000FF starting color
green with 927 counts converted to #adff2f instead of #008000 starting color

Other than than, very impressive how the color snapping performs!

Looking at the hits and misses for counts ≥ 10 were the two palettes outperform each other, I propose adding the following to mine, for a total of 49 colors (so close to 50, we'll get there – but less than Crayola 64!).

I've also simplified the table to just include a single hex value (I wonder if the mz_hex was used instead of the css_hex last time to compute the snaps – I've preserved a few client-side proposed mappings in the comment column, but think we should snap only on the css_hex values).

Please prototype with the following table:

new palette.txt confirm round trip the earlier work
dev tiles build with line color snapping, storing new css color name into new colour_name property

order	css name	css hex	comment
00	`white`	`#ffffff`
01	`yellow`	`#ffff00`
NEW1	`khaki`	`#f0e68c`
24	`gold`	`#ffd700`	Client-side `#ffcc11`
02	`orange`	`#ffa500`
16	`sandybrown`	`#f4a460`	(light orange)
10	`tan`	`#d2b48c`	Client-side `#cd853f`
NEW2	`goldenrod`	`#daa520`
17	`peru`	`#cd853f`	Client-side `#d9680c`
11	`brown`	`#a52a2a`	Client-side `#90713a`
33	`tomato`	`#ff6347`
NEW3	`orangered`	`#ff4500`
03	`red`	`#ff0000`
NEW4	`darkred`	`#8b0000`
34	`crimson`	`#dc143c`
18	`lightcoral`	`#f08080`
28	`hotpink`	`#ff69b4`
04	`fuchsia`	`#ff00ff`	AKA magenta, client-side `#f20884`
20	`violet`	`#ee82ee`
23	`plum`	`#dda0dd`
19	`pink`	`#ffc0cb`
NEW5	`mediumpurple`	`#9370db`
33	`darkorchid`	`#9932cc`
NEW6	`indigo`	`#4b0082`
05	`purple`	`#800080`	Client-side `#991199`
06*	`blue`	`#0000ff`	Client-side `#0000d3` (nearer medium blue)
NEW7	`navy`	`#000080`
29	`royalblue`	`#4169e1`
30	`dodgerblue`	`#1e90ff`
31	`steelblue`	`#4682b4`
25	`teal`	`#008080`
26	`lightteal`	`#0099aa`
NEW8	`mediumturquoise`	`#48d1cc`
21	`lightblue`	`#add8e6`
07	`aqua`	`#00ffff`	AKA cyan, client-side `#02abea`
07	`aquamarine`	`#7fffd4`
22	`darkseagreen`	`#8fbc8f`
NEW9	`mediumseagreen`	`#3cb371`
32	`olivedrab`	`#6b8e23`
35	`yellowgreen`	`#9acd32`
NEW10	`lime`	`#00ff00`
08*	`limegreen`	`#32cd32`	Client-side `#79b51d`
NEW11	`lightgreen`	`#90ee90`
09	`green`	`#008000`
27*	`gainsboro`	`#dcdcdc`
12	`silver`	`#c0c0c0`
13	`grey`	`#808080`
14	`darkgrey`	`#404040`
15	`black`	`#000000`

zerebubuth commented 7 years ago

I'm confused how:

blue with 1344 counts converted to #8a2be2 instead of #0000FF starting color

green with 927 counts converted to #adff2f instead of #008000 starting color

That's just me failing to parse the X11 rgb file correctly. There's a space in the colour names for "blue violet" and "green yellow" which I wasn't handling, so they got assigned to "blue" and "green" respectively.

I'm moving to a library for doing these, so hopefully that handles things a bit better.

zerebubuth commented 7 years ago

Latest palette.txt, and the top 20:

count	colour	MZ colour	hex colour
1344	blue	blue	`#0000ff`
927	green	green	`#008000`
854	red	red	`#ff0000`
610	black	black	`#000000`
533	`#79b51d`	olivedrab	`#6b8e23`
311	`#FF0000`	red	`#ff0000`
275	`#0000FF`	blue	`#0000ff`
272	yellow	yellow	`#ffff00`
262	`#CD853F`	peru	`#cd853f`
202	`#000000`	black	`#000000`
194	`#FFFF00`	yellow	`#ffff00`
184	`#a2bd31`	yellowgreen	`#9acd32`
164	`#ff0000`	red	`#ff0000`
163	`#ED1C24`	crimson	`#dc143c`
162	`#008000`	green	`#008000`
148	`#f77921`	tomato	`#ff6347`
147	`#005266`	teal	`#008080`
143	orange	orange	`#ffa500`
116	`#BB0022`	crimson	`#dc143c`

nvkelso commented 7 years ago

Looks great!

nvkelso commented 7 years ago

I see this on dev and it's mostly working :)

We can followup with refinements in another issue in the next milestone.

tilezen / vector-datasource

Calculate colour_name property for transit lines #1190