Point color by type

[Friptick]

Is it possible to give points different colors according to their type? Anything to distinguish between types (created with Osmand, by the way) would be so useful - if only attributing a different color randomly to each type of point. At present I find that GpsPrune is not very useful for POIs because of this issue.

In Settings > Set colours there is the mysterious tip "A point colourer can give track points different colours". Does the feature exist already?

If not, consider this is a grateful feature request!

Thank you for your work.

[activityworkshop]

The important word there is giving track points different colours. So yes, that feature exists already, and if you open up the dropdown box right there it will let you select which criteria you want to use to give the track points different colours (eg by file, by segment, by altitude, ....).

But I'm guessing that this isn't exactly what you want. When you say "useful for POIs", I think you're talking about waypoints (points with names). These are drawn differently from track points, using the waypoint icons specified in Settings -> Set display options in the section "Waypoint icons". So you can choose one icon and its size which will be used to draw all the waypoints. Also usually the waypoint name will be drawn for each waypoint too.

So it sounds very related to the request to render waypoints in a custom way according to the "waypoint type" field, see here: https://github.com/activityworkshop/GpsPrune/issues/3 - this has been an open request for years (!). Please correct me if I've misunderstood, or feel free to expand your ideas if I haven't ;)

[Friptick]

Thanks so much for the thorough reply! Appreciated!

Yes, I do mean waypoint, you're right.

And the request in #3 is very similar, yes - unfortunately I didn't see it when I searched. To be honest I'm not clear what the other user is asking for exactly, but if you understood it as "color by type" then that is right.

Specifically, Osmand marks up its POIs in the default favourites.gpx like this: <wpt><type>Foo</type></wpt>. Alongside <type> it also has a nonstandard element, <extensions><osmand:color>#f0f0f0</osmand:color></extensions> by which the <wpt> color is specified. In practice, you can give types "Foo" and "Baz" different colors in the interface, and they will be marked up accordingly. And I believe that on their roadmap they intend to make types properly associable with specific colors and icons and maybe even image files. Which would be very logical - because nobody wants to style waypoints individually.

What I believe would be a great first step is just to give a different color to each different type of waypoint. The specific colors are unimportant - just a simple algorithm to attribute a new color to each new type string found. This would make the type markup instantly useful, where right now it is effective ignored.

Thanks for consideration and thanks for your work on this project.

[activityworkshop]

You're correct, what you're asking for isn't exactly the same as number 3. That request was for using different custom symbols, and the problem was (or still is) that there's no clear mapping from a waypoint type (which could be anything, in any language) to a set of symbols. And the complexity of offering the user the ability to supply their own custom icons is (I believe) way beyond what most users want. That's why the current solution uses one single selectable symbol.

But your suggestion is slightly different, and indeed it would be possible to paint the chosen symbol (like the white bit of the 'ring' symbol for example) with different colours for different waypoint types. It would still be fairly arbitrary what the colours mean though, which could be very confusing.

For example, let's say the colouring would work in a similar way to the colouring of track points by segment. The first waypoint type encountered (eg "museum") would be given (for example) red and the second waypoint type ("campsite") would be (for example) blue. Great. So then you get used to the fact that museums are red and campsites are blue, but then you load a different track in which the first waypoint is a campsite, or even just delete a few museums from the beginning of the track. Now you're confused because all the campsites are now red and the museums are blue! And there's (currently) no way to display a colour key to resolve the confusion.

One solution to the ordering could be to collect a list of all the waypoint types in the track and sort them alphabetically, so campsites always get an earlier colour than museums (because c < m) even if the first waypoint in the track is a museum. But still adding or removing waypoints of other types (eg "basketball") will shift the colours used.

Maybe in your case it's easy because you always use only two or three waypoint types? But I'm not sure how it would work in general. I like the idea though, and I can see the benefit you're trying to get.

(By the way I don't think GpsPrune should use the <extensions><osmand> tags, it should only use the <wpt><type> or possibly <wpt><sym> instead). I agree that nobody wants to style their waypoints individually, or if they do then they don't use GpsPrune, they use a fully-featured GIS software instead.

Another smaller issue is the dialog for configuration - currently the colours (both static definitions and specification of the colourer for track points) are all in Settings -> Set colours, which is completely separate from the display options in Settings -> Set display options (which currently includes the waypoint icons). If the waypoint icons then get different colours too then the configuration panels will need to be reorganised - and I'd like to avoid tabs if possible. Of course that's not a big deal, just the settings which are related to each other should be kept close to each other if possible.

[Friptick]

Yep @activityworkshop, I agree with your conclusions.

The downside of arbitrary colors did occur to me after suggesting it. You're right it's not worth implementing in that way, after all.

Absolutely agreed that <extensions> tag is a terrible idea. Reminds me of inline styles in HTML. Presentation config should be kept out of markup. That goes for Osmand too - but I believe they might be moving away from it.

So yes, realistically the only way is going to be some kind of database table with a color selector. You would know the best exact approach to take. I understand you have other priorities too. But as long as the type element exists in the spec, it does seem to me to be a necessary feature one day.

Thanks again for your work on this project. Very appreciated.

[activityworkshop]

Here's a first experiment to see whether it could work or not: I have no idea what types you've got in your data, or whether you create those types deliberately or just download files with predefined types. So this just tries to derive a colour and hopes that the colours for the different types which you use are different enough to be distinguished by eye. With this approach though, your "campsite" type will always have the same colour, no matter what other data is present in the current track.

Of course, if you have a huge variety of types in your data, then inevitably some of them will look identical or indistinguishable - that seems unavoidable.

If you can point me towards some example data (doesn't have to be real, just kind of representative) then I can see how my approach works for that.

[Friptick]

This looks great.

I suppose there are three potential requirements for the colors:

1. be distinguishable
2. not change randomly
3. be customizable

As you say, 1 depends on not having too many types, independent of the implementation. So this problem is for the user.

A feature for 3 would of course be nice, to get the same colors on both platforms. But if you can ensure 2 for much less work, then that is definitely progress. For example, Osmand can now style types with icons too - so the potential issue of having to remember that "yellow" in Osmand is the same as "red" in GpsPrune could be ameliorated by using icons in Osmand. Or perhaps vice-versa in the future. But I'm not clear on how you plan to make the colors immutable if they are attributed on the fly?

Anyway, yes, I have about 10 types, for things like "restaurant" and "hotel" and "cultural attraction". And literally thousands of points. I noticed in Osmand that when the points are very small, you can only distinguish easily between about 10 colors. But that is probably more than enough for many users.

[activityworkshop]

I still haven't found any "standard" or "typical" waypoint types, so if anyone can provide links to what's commonly used, I'd appreciate it.

I recorded a selection of waypoints on my GPS receiver and gave them different symbols, but in the GPX file these symbols come out in the <sym> tags - in my case they're written in English, in mixed-case, like "Campground", or "Picnic Area", or "Beach". But the <type> tags are missing, so this symbol colouring wouldn't be much use to me (even if I did take the trouble to select a symbol type for each waypoint I record, which I usually don't!). I also created a file with a variety of <type> tags and transferred it to the receiver, but these types were just ignored and weren't displayed anywhere, and of course the points were all given the same default symbol.

So if GpsPrune's symbol colouring only makes use of the <type> tag and not the <sym> tag, then it would be of very very limited use to me personally, maybe for OsmAnd users it would be greater? I couldn't find any web-based gpx viewers which made use of the types either, which makes me wonder how much they're actually used.

This screenshot shows an example of 11 different waypoint types and the colours they are given - arbitrary but consistent (not random). It ignores the upper/lower case, which is why "Camp" and "camp" are treated identically, as are "restaurant" and "Restaurant". But obviously "Camping" is different from "camp", and "campsite" and "camp site" would also be treated independently.

So if I take care to call all my campsites in exactly the same way, then they would all appear in the same colour, regardless of what other waypoint types are present or in what order. In this case, "cinema" and "museum" are quite close to each other and probably quite difficult to differentiate unless the waypoints are close to each other. Also "hotel" and "station" are quite close, so depending on your data maybe this is a good distribution or maybe not. If you only ever care about restaurants and hotels and stations then maybe not, but if you only ever deal with operas, museums and campsites then it could be perfect.

So one idea is that you could look at these colours and decide that you don't like them, then you could press a "shuffle" button and all the colours would change to a new set. Obviously this set would be saved in your settings so it would be consistent unless you change it again. So if you take the types which you often use, you can shuffle until you get a set which works for you. Here are some more examples:

In this next one, "cinema" and "museum" are no longer close together, and "hotel" and "station" are quite different too, but now you've got restaurants and bars quite close together. Or, depending on your shuffle, campsites and cinemas. Or restaurants and operas. Or restaurants, hotels and stations. So you could shuffle until you're happy (for the types you use) and then save your settings, and from that point on the colours stay fixed.

Another possibility, which starts to demand more from the user, is to allow "exceptions", so you could manually specify a particular colour for your type "hotel" and just use automatic colours for the other types. So you wouldn't be forced to define a separate colour for all of your types, but just the one(s) you want to be non-automatic. I'm not sure I like asking the user to do this, but maybe there are a few users who actually want to do that work and wouldn't be happy with just shuffling.

Thoughts?

[Friptick]

Looks great! Some thoughts.

Here is the doc: https://www.topografix.com/GPX/1/1/#type_wptType

So type is defined quite clearly as "type (classification) of the waypoint", while sym is defined enigmatically as "symbol" - which to me also seems to confusion semantics and presentation (it is not for the markup to define what the point looks like visually). So: I'd say Osmand is producing the right markup in this case. What about mapping type and sym to the same meta data so the result is the same?

Your idea of shuffling sounds okay but, as you hint, an exceptions-based functionality may be a bit laborious. If there will have to be a persistent database table, why not just allow users to pick all the colors? My original idea was for a quick fix, but if an actual color picker is feasible, that's the ideal solution! Or even just text fields for inputting colors in hex format would be great. Assuming the logic for random allocation is not too hard, then that could be left as the default. At this point there would really be nothing left to improve!

Thanks again.

[activityworkshop]

What about mapping type and sym to the same meta data so the result is the same?

Yeah, that's a good idea. I had similar thoughts about the Comment and Description fields, to duplicate one to the other if only one of them is present.

why not just allow users to pick all the colors?

Because it's more work required by the user. Then we're back to my earlier claim that "nobody wants to style their waypoints individually, or if they do then they don't use GpsPrune". That may not be true for all users but I believe that most will be happy with less than that and don't want to have to configure everything individually. Plus I don't think we can assume that the user knows all the types they're going to use in the future.

I can see various possible levels of this, from the current position (no differentiation at all) up to completely manually configurable:

• Level 0 (current state of 21.1) - all waypoints displayed identically, no configuration options
• Level 1 (optional but automatic) - user can choose no colouring (as before) or automatic colouring (no options)
• Level 2 (automatic with shuffling) - as described above, with the option to shuffle until satisfied
• Level 3 (automatic with exceptions) - most colours are automatically assigned unless the user manually specifies a colour for a type
• Level 4 (manually defined) - user has to specify all colours for all types, others are left white

You write "an exceptions-based functionality [Level 3] may be a bit laborious... why not just allow users to pick all the colors [Level 4]?". I feel that Level 4 is actually more laborious than Level 3, but there's probably little difference between them. So the question is where do most users live on this scale? Currently I'm at level 0, but if it turns out to be useful I'd probably be happy with 2 (I don't think level 1 helps anybody). For those users living at 1 or 2, they wouldn't be happy with level 4. For those users really requiring level 4, they wouldn't be happy with 2 but could be happy with 3, so maybe that is our "sweet spot"? So those who really want to define all the colours can do so, but those who don't want to aren't forced to.

I'm not sure how the settings dialog will work yet, but this looks like it's going to be a good addition to version 22. Thanks for the inputs.

[Friptick]

This is great. I'd be happy with any of these. And delighted with Level 3 or 4.

And I do think even Level 1 is helpful. After all, points are not given types by default. If the points are marked up with different type or sym, then we can assume the user wants to distinguish between them. That why it seemed to me like an almost bug-level oversight for them all to be styled identically.

BTW I'm on Ubuntu LTS, so only at V19.2 right now. Will have to hope the upcoming 22 LTS has a newer version.

[activityworkshop]

If the points are marked up with different type or sym, then we can assume the user wants to distinguish between them.

Maybe, maybe not - it's possible that the user doesn't care about waypoint types (and they will be ignored by their GPS receiver anyway) but it just so happens that the gpx file (or kml/kmz/whatever) which they downloaded from the internet uses types or symbols or icons for some reason.

Ubuntu's release schedule is perhaps off-topic for this issue, but yeah 19.2 is from way back in 2018 so you're missing out on all the features and changes from the 20. and 21. releases. Even Debian Stable (famous for being slow to include new versions of software) got version 20 in early 2020 and currently has the latest 21.1.

If I were running Ubuntu LTS, I'd try downloading the .deb from Debian (bullseye-backports) and installing it with dpkg. This is usually a terrible idea but in the case of GpsPrune's dependencies there shouldn't be any issues. Or do what Windows users do and just download the latest jar of GpsPrune and run it with java -jar gpsprune.jar.

Ubuntu "Jammy" will (finally!) contain GpsPrune 21.1, so you would likely have to wait until April 2024 for your waypoint colouring feature if you stick with the LTS-provided packages.

[Friptick]

Useful, thanks. I'm usually happy to accept outdated LTS packages - no bugs, zero maintenance.

just download the latest jar of GpsPrune and run it with java -jar gpsprune.jar

This sounds like the best compromise.

Doing it now, can confirm 21.1 works great! :)

Can't wait for the new feature. Thanks again for all your work on this project.

[activityworkshop]

A draft version of the settings dialog (for version 22) can be seen here: https://gpsprune.activityworkshop.net/development.html (Level 2, no exceptions)

[Friptick]

Very nice! Looks great, can't wait to try it out. :)

[activityworkshop]

I'll close this, but if you have further suggestions they can go in a separate issue :)

[Friptick]

Great! Looking forward to trying it, will provide feedback if required. Thanks for your work.

[Friptick]

Can confirm that version 22.2 is now here:

Level 2 (automatic with shuffling) - as described above, with the option to shuffle until satisfied

Is that correct? Much better than nothing (thanks!) but will be great to have the upgrade:

Level 3 (automatic with exceptions) - most colours are automatically assigned unless the user manually specifies a colour for a type

Is it still on the roadmap?

[activityworkshop]

That's right, 22.* has the shuffling, but without exceptions. Version 23 will have exactly the same.

Manually specifying exceptions isn't currently high on the priority list, unless the demand is high ;) so far the demand doesn't seem to be very high. If you've already done some thinking about this then maybe you can suggest how you think it should work. Maybe in a new issue, to avoid re-opening this one.

[Friptick]

so far the demand doesn't seem to be very high

Generous of you to put it so politely. :)

FYI (beware incoming rant) here's the annoying issue I'm (still) facing. My use case: see POIs by type, on both desktop and mobile (here Android). As far as I can tell, Osmand and Guru Maps (and possibly Orux) are the only Android apps that allow any kind of customization of waypoints by type. No idea why, it's a part of GPX spec, it's super handy in theory and in practice, to me it seems the most obvious thing in the world, but hey, that's the way it is. Next, Guru Maps and Orux do not allow syncing of maps via syncable GPX files. This is true of almost all mapping apps: it's always a choice between a cumbersome manual import-export feature or miscellaneous commercial cloud solutions ("Sign in" etc). Which leaves Osmand. And Osmand's waypoint-type feature is only semi-functional! It uses the GPX <extensions> namespace to attach colors and icons to the individual waypoints, so that changing a waypoint's "group" (type) leaves its color unchanged! Exasperating. There's an issue open, but apparently demand for resolution is not high. ;) End of rant.

Anyway, since GPSPrune cannot (and should not need to) read or write Osmand's <color> extension, the only way forward, if I want to see the same rough colors per type at both ends, is to accept GPSPrune's color suggestions and then change them to something similar in Osmand (it is capable of rewriting waypoints en masse via a bulk operation, go figure).

So I come to the point!

Level 2 (automatic with shuffling) - as described above, with the option to shuffle until satisfied

There's a small but annoying issue. This works in principle, but the colors proposed are currently not quite different enough! Assume a roster of 5 waypoint types. Of the 10 color permutations proposed, 7 contain two colors that are all but indistinguishable. See capture for example with dummy GPX.

BTW that example is even more extreme than with my real-life GPX. Could it be that you based the color values on the text string of the type name? Making them completely random would surely be an improvement on that. Best of all would be an algorithm that distributes the hues evenly, but maybe that's terribly complicated!

Whatever can be done by way of fine-tuning would be most appreciated.

[activityworkshop]

Could it be that you based the color values on the text string of the type name?

Given that any points from any files with a text string of "Restaurant" need to have always exactly the same colour regardless of filename, coordinates, timestamp or other points present in the file, can you imagine a way in which that colour was not based on the text string of the type name?

If the colours were completely random they'd change for every point in every file, and I don't think anybody would find that useful, let alone "an improvement".

of the 10 permutations ... 7 contain two colors that are all but indistinguishable.

then what's wrong with choosing one of the other 3, for which the colours are distinguishable?

[Friptick]

There's something I don't understand. Earlier you said

Obviously this set would be saved in your settings so it would be consistent unless you change it again

I.e. the color values are persistently stored and not being generated on the fly from the the type strings. Is that incorrect?

Because if it's saved in my settings, why can't the original color choices be generated randomly? Or by a function that distributes them according to the number of types in the loaded file? And indeed if it's in my settings, could the generated value not just be editable in the settings file? Like a manual override to avoid complicating the UI for users that don't care.

Feels like I'm still not understanding something very basic here! Thanks for your patience.

[activityworkshop]

Here's an improvement coming with version 23 to handle your "Type 1", "Type 2", "Type 3" use case: I was assuming until now that the type names were meaningful and therefore quite different, but yours are identical apart from the last character, each one different by only 1. This was putting pressure on the colour generation and that's why your colours were so similar. This should be an improvement.

the color values are persistently stored and not being generated on the fly from the the type strings. Is that incorrect?

No, that's not correct. No colour values are persistently stored, and all colours are generated on the fly from the type strings. If the colour values were persistently stored, then your second computer would show different colours, and I understood it to be a requirement that colours are always consistent on all devices. Therefore only the number from 0 to 9 is stored, and this is used to generate the colours. To synchronize your devices, all you need to do is select the same number from the slider, and then all colours will be identical, with no randomness. And importantly, they will also be identical for point types which GpsPrune has never seen before.

a function that distributes them according to the number of types in the loaded file?

Again, that would break the requirement that a campsite always has the "campsite" colour - if I load file 1 then campsites are green, but if I load file 2 then campsites are orange. I understood that consistency is very important as the user will get used to campsites always being a specific colour, no matter which file they're loaded from.

could the generated value not just be editable in the settings file?

That would be possible, with some development effort. But explaining how to use it would be rather awkward, and using it would be awkward too.

[Friptick]

This looks like a decent improvement.

I was assuming until now that the type names were meaningful and therefore quite different, but yours are identical apart from the last character

In fact they were just dummy names but it's a good idea to allow for one-character offsets because of the existence of numbers. Big difference between "2StarHotel" and "3StarHotel"!

all colours are generated on the fly from the type strings

Yes I get it now. In fact you're just storing the "salt" (0-9). And you can't take into account the number of types because that might throw things off from file to file.

To synchronize your devices, all you need to do is select the same number from the slider, and then all colours will be identical, with no randomness.

This may be the misunderstanding about my own use-case. My other device does not have GpsPrune! It's an Android. And Osmand stores POI information solely in the GPX markup as a namespaced extension:

    <wpt lat="10.00" lon="10.00">
    <name>The Mediocre Inn</name>
    <type>3StarHotel</type>
    <extensions>
      <osmand:color>#eecc22</osmand:color>
    </extensions>

In other words they're doing it wrong and you're doing it right, IMO. Presentation data should not be hardcoded in bespoke markup like that, it's completely unportable. (Have raised the issue with them.)

So until they fix that, the best I'm going to get is a semi-solution. I.e., changing the colors in Osmand to agree with whatever your script comes up with. (I've already written a Python script that adds the Osmand color markup to POIs on the basis of their type. What a PITA.)

I must admit that your reticence to allow editing of the color values is a bit mysterious to me. After all, you're already storing color data that's only marginally more minimal than, say, #ff0000. And if it's about UX, surely these days everyone understands at a glance what a color-picker is? Or perhaps it's also hard to do in the Java SDK? I personally don't care much about the actual colors as long as they're clearly distinguishable but I can see other users getting frustrated when they see color options and then find they can't just pick the one they want.

Anyway, it's your call of course, and this is very definitely an improvement as it stands. Thanks.

[activityworkshop]

Earlier you wrote:

I'd be happy with any of these. And delighted with Level 3 or 4.

So although level 3 might come in the future (but not for version 23), I'm going to assume that you're "happy" with level 2 as it is.

it's a good idea to allow for one-character offsets because of the existence of numbers. Big difference between "2StarHotel" and "3StarHotel"!

Those don't only differ by the last character. Your hotels were/are handled properly already, as far as I can tell.

This may be the misunderstanding about my own use-case. My other device does not have GpsPrune!

I don't think it was a misunderstanding, perhaps I just used the word "your" incorrectly. If in general it can be said that points of type "campsite" must always have a consistent colour, then that applies also to a user with two computers (desktop/laptop, or work/home) with GpsPrune running on both. They should use the same colours, with no randomness and no sensitivity to which point types are present in the current file.

After all, you're already storing color data that's only marginally more minimal than, say, #ff0000. And if it's about UX, surely these days everyone understands at a glance what a color-picker is? Or perhaps it's also hard to do in the Java SDK?

I don't quite understand that first sentence, and it sounds like you haven't used the existing colour-pickers in GpsPrune. If you had, then you'd know that GpsPrune does provide colour-pickers, and does save colour data in the settings. It just doesn't do that for special exceptions for specific point types which behave differently from other point types.

The "reticence" to make the point type colouring more complicated than it already is is just based on the number of people asking for level 3 rather than level 2. For those people, is it justified to make that dialog more complex (and maybe more confusing) for other users? It's not just a colour-picker, it's an explanation of what exceptions are currently defined, it's making clear why some point types in that panel don't change colour when you change the slider, it's making it possible to edit the exceptions, add new exceptions, delete exceptions. The actual saving and loading of the exceptions would be fairly trivial I think, but is the demand there for that? Based on the number of issues raised about it (and closed about it), I'd say perhaps not. Maybe you're right, maybe there are many "frustrated" users who don't like any of the colour sets offered, but I'm not hearing from many such people ;) So I see level 2 as a compromise which should keep most people happy.

[Friptick]

Your hotels were/are handled properly already, as far as I can tell

Believe it or not, none of my actual categories feature hotels or campsites or any other form of accommodation. :) Trust me, the last thing I want is for you to spend your time designing bespoke features for me. That's why I always try to make my examples as generic as possible.

If in general it can be said that points of type "campsite" must always have a consistent colour, then that applies also to a user with two computers (desktop/laptop, or work/home) with GpsPrune running on both

Agreed. My point was that this semi-arbitrary system is a bit fragile when any other software is being used with the same GPX data, because it means the other software must have a color-picker. Imagine the other software works the same as GpsPrune, with a dictated choice of colors - there will be no way for the user to make the colors agree. Anyway, it's something of a detail, yes. In fact, personally I've decided to leave unchanged the colors "on the other end" because I'm familiar with them, and just accept 2 sets of colors. It's not so bad, but neither is it ideal.

The "reticence" to make the point type colouring more complicated than it already is is just based on the number of people asking for level 3 rather than level 2.

Of course. But what if your system were actually the "more complicated"? That's my contention. As a programming exercise the algorithm is pretty cool but I'm still having trouble seeing how its UX can be less easy to grasp than a generic color-picker (or just a text field and a hex value), which can all be entirely optional anyway because the default can be "same color". As you say, most users don't even care. The problem is that the ones that do will find themselves constrained. And all to avoid a semi-hidden color picker which would be easy to implement anyway? It's your project, this is just my two cents.

There's a small philosophical issue at stake here too. Personally, as much as I hate Apple I think they're right that most users don't know what they want until you give it to them. And therefore software can be designed rationally as well as just empirically. Does this mean you can now ignore me because I don't know what I want either? Possibly. :)