Store custom presets in OSM user profile

tordans commented 2 years ago

Context

The OSM api allows to store preferences in a (shared) store per user.

UseCase

I would love to add more custom presets to my app, but doing so via the phone is just too much clicking
I would love to share custom preferences between devices (Mac, iPad, iPhone)
As a beta test user, I have to reinstall and reset my settings more often, which means that the fact that adding presets manually on the phone is even more of an issue, since I have to make it over and over…
Same goes for the list of custom image layers; right now I have to copy paste the URLs from desktop to mobile to add the to the UI, which is quite time consuming

Idea

Store the custom data for background layer and custom presets in the osm user profile. Use a YAML or JSON format. Reset this store whenever there is an issue and recreate it from scratch, so the app always rund fine, so matter what the API delivers.

This will solve the multi device issue and the beta-test-reset issue.

For adding presets more easily: Once they are stored externally I can use the Mac app with keyboard and mouse to add presets, and use them on mobile.

bryceco commented 2 years ago

I believe I looked into this once before and found that the key/value store for preferences is limited to 256 character values, so using a JSON based format to store multiple presets isn't scalable. There would have to be some trickery with splitting it across an array of keys (key1, key2, key3, ...) and then combining them together to build the final JSON.

What would be very helpful is if you could provide an example of all the presets that you typically like to create, so I'll have an idea of how much data you're typically dealing with.

tordans commented 2 years ago

I believe I looked into this once before and found that the key/value store for preferences is limited to 256 character values, so using a JSON based format to store multiple presets isn't scalable. There would have to be some trickery with splitting it across an array of keys (key1, key2, key3, ...) and then combining them together to build the final JSON.

@mmd-osm I suspect you will know more about this from the top of your head; which would be helpful. In case it is a 256 char limit, so you see a possibility to change that in the rails port or is that baked in the current system because all api tags are the same and therefore share the same limits for values?

What would be very helpful is if you could provide an example of all the presets that you typically like to create, so I'll have an idea of how much data you're typically dealing with.

Will do…

mmd-osm commented 2 years ago

Key/Values are actually limited to 255 characters (like all key/value pairs elsewhere on the Rails port), see https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/user_preference.rb#L20

I don't see this could be changed on short notice. However, you can use similar tricks like overpass turbo: compress your payload, and use multiple key/value pairs to store that information.

tordans commented 2 years ago

What would be very helpful is if you could provide an example of all the presets that you typically like to create, so I'll have an idea of how much data you're typically dealing with.

1. shared custom presets for tags

There are two kind of presets, that I would like to add:

Full new presets
Custom modifications for presets

(See comments inside the JSON for ideas about that.)

The list below is what I have right now. I have a few more that I did not bother to add, yet. The parking:lane presets would be a bit longer, since this schema has many tags. It is not in the official preset-list yet, since it would require custom fields to work properly. But since those presets are for myself only, I can live with the non-ideal usability.

About the JSON format: We should have a look at the id-tagging-schema again; but the schema below might be an easier first start. I think about it as a "input with dropdown only" schema, so no custom fields, just the default field that GoMap has now. One could limit the number of entries accepted in those arrays to 10 like the UI does right now, if that makes it easier to integrate it with the current UI.

[
  {
    "name": "Garbage collection ramp", // the name of the stand alone preset
    "extend": [], // meaning, this is a stand alone preset
    "tags": [
      {
        "key": "kerb",
        "defaultValue": "flush", // meaning, this value will be selected whenever I select the preset
        "values": ["flush"]
      },
      {
        "key": "operator",
        "defaultValue": "BSR",
        "values": ["BSR"]
      },
      {
        "key": "ramp",
        "defaultValue": "yes",
        "values": ["yes"]
      }
    ]
  },
  {
    "name": "Tree ref", // the name of the label for this group of tags inside the existing preset
    "extend": [{"key": "natural", "values": ["tree"]}], // meaning, this preset extends an existing preset; is not searchalbe
    "tags": [
      {
        "key": "ref",
        "defaultValue": null, // no default value
        "values": [] // no suggestions
      },
      {
        "key": "start_date",
        "defaultValue": null,
        "values": []
      }
    ]
  },
  {
    "name": "Bike parking position",
    "extend": [{"key": "amenity", "values": ["bicycle_parking"]}],
    "tags": [
      {
        "key": "bicycle_parking:position",
        "defaultValue": null, // no default value
        "values": ["sidewalk", "lane", "… a few others or just the taginfor search…"]
      }
    ]
  },
  {
    "name": "Tree barrier",
    "extend": [{"key": "barrier", "values": []}],
    "tags": [
      {
        "key": "barrier",
        "defaultValue": null, // no default value
        "values": ["collision_protection"] // this would be tricky, I want add a suggestion to an existing tag "dropdown"; or to duplicate the dropdown to give easy access to this tag; alternatively I could make this a stand alone preset "Barrier collision protection" – which would probably be the best idea and easiest to implement TBH.
      }
    ]
  },
  // {
  //   "name": "Presets for all the different ",
  //   "extend": [{"key": "highway", "values": ["residential", "secondary", "… and a few others …"]}],
  //   "tags": [
  //     {
  //       "key": "parking:lane",
  //       "defaultValue": null,
  //       "values": ["… a few values …"]
  //     },
  //     {
  //       // … see https://wiki.openstreetmap.org/wiki/Key:parking:lane, there are a few of those
  //     }
  //   ]
  // },
]

2. shared custom editor layer index entries

The other thing are editor layer entries, that I don't want to add to ELI, since they are not ready for every user, but I would like to use them myself.

The JSON only shows the properties part from – for example – https://github.com/osmlab/editor-layer-index/blob/gh-pages/sources/europe/de/Berlinaerialphotograph2021.geojson?short_path=2bfb637, because those don't need a geometry IMO. They are added by people for their local needs, so they know when to use them…

[
  {
    "name": "Parkraumkarte", // https://supaplexosm.github.io/strassenraumkarte-neukoelln/?map=parkingmap
    "type": "tms",
    "url": "https://supaplex.uber.space/micromap/tiles/parkingmap/{z}/{x}/{y}.jpg",
    "attribution": {
      "text": "OpenStreetMap; Bordsteinkanten: OpenStreetMap und Geoportal Berlin / ALKIS",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "Straßenraumkarte", // https://supaplexosm.github.io/strassenraumkarte-neukoelln/?map=micromap
    "type": "tms",
    "url": "https://supaplex.uber.space/micromap/tiles/{z}/{x}/{y}.jpg",
    "attribution": {
      "text": "OpenStreetMap; Bordsteinkanten: OpenStreetMap und Geoportal Berlin / ALKIS",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "Neukölln Debug Map", // https://supaplexosm.github.io/strassenraumkarte-neukoelln/?map=debugmap
    "type": "tms",
    "url": "https://supaplex.uber.space/micromap/tiles/debug/{z}/{x}/{y}.jpg",
    "attribution": {
      "text": "OpenStreetMap; Bordsteinkanten: OpenStreetMap und Geoportal Berlin / ALKIS",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "Straßenbefahrung 2014", // https://github.com/codeforberlin/mapproxy-config/tree/master/demo_links#stra%C3%9Fenbefahrung-2014
    "type": "tms",
    "url": "https://mapproxy.codefor.de/tiles/1.0.0/strassenbefahrung/mercator/{z}/{x}/{y}.png",
    "attribution": {
      "text": "Geoportal Berlin / Straßenbefahrung 2014",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "Bäume Berlin", // https://github.com/codeforberlin/mapproxy-config/tree/master/demo_links#baumbestand-berlin---stra%C3%9Fenb%C3%A4ume-und-anlagenb%C3%A4ume-mit-und-ohne-datenbankeintrag
    "type": "tms",
    "url": "https://mapproxy.codefor.de/tiles/1.0.0/baumbestand_0_1_3_4_merged/mercator/{z}/{x}/{y}.png",
    "attribution": {
      "text": "Geoportal Berlin / Baumbestand Berlin - Straßenbäume und Anlagenbäume mit und ohne Datenbankeintrag",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "ALKIS Berlin", // https://github.com/codeforberlin/mapproxy-config/tree/master/demo_links#baumbestand-berlin---stra%C3%9Fenb%C3%A4ume-und-anlagenb%C3%A4ume-mit-und-ohne-datenbankeintrag
    "type": "tms",
    "url": "https://mapproxy.codefor.de/tiles/1.0.0/alkis_30/mercator/{z}/{x}/{y}.png",
    "attribution": {
      "text": "Geoportal Berlin / ALKIS Berlin (Amtliches Liegenschaftskatasterinformationssystem)",
      "required": true
    },
    "max_zoom": 20
  },
  {
    "name": "Brandenburg GeoBasis-DE/LGB (2017-2020) / DOP20c", // https://github.com/osmlab/editor-layer-index/pull/933/files
    "type": "wms",
    "url": "https://isk.geobasis-bb.de/ows/dop20c_wms?FORMAT=image/png&TRANSPARENT=TRUE&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=bebb_dop20c&STYLES=&SRS={proj}&WIDTH={width}&HEIGHT={height}&BBOX={bbox}",
    "attribution": {
      "text": "GeoBasis-DE/LGB 2020 / Digitale farbige Orthophotos aktuell",
      "required": true
    },
    "max_zoom": 20
  }
]

shared spec for those formats

IMO, this kind of spec would be suited to share it between editors (native and web). For this, we might want to extract the JSON Schema once it is tested.

tordans commented 2 years ago

About saving the data:

I can have a list at all key/values stored in my user profile via https://api.openstreetmap.org/api/0.6/user/preferences (username, password from the website).

I have quite a bit of stuff lying around there (see below)…

But it looks like, it should be OK to make every entry of the top level array a tag. For example: custom-editor-preset-v1[some_hash_for_uniquenes]=…

Same for the custom ELI: custom-editor-layer-v1[some_hash_for_uniquenes]=…

Having the schema version "v1" in there is probably good to know what to expect when parsing. And maybe use a random hash so one does not need to rewrite the array index every time?

~~IMO it should be enough to just stringify the JSON for the value(?)~~. Update: Maybe not, 255 is not a lot…

> JSON.stringify(a)
'{"name":"Straßenbefahrung 2014","type":"tms","url":"https://mapproxy.codefor.de/tiles/1.0.0/strassenbefahrung/mercator/{z}/{x}/{y}.png","attribution":{"text":"Geoportal Berlin / Straßenbefahrung 2014","required":true},"max_zoom":20}'

> JSON.stringify(a).length
232

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="OpenStreetMap server" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
  <preferences>
<preference k="diary.default_language" v="en"/>
<preference k="gps.trace.visibility" v="public"/>
<preference k="mangroveidentity-combined-0" v="{&quot;crv&quot;:&qu…;,&quot;metadata&quot;:&quot;Mangrove private key&quot;}"/>
<preference k="mangroveidentity-combined-length" v="1"/>
<preference k="mapcomplete-current-open-changeset-berlin_emergency_water_pumps" v="120339118"/>
<preference k="mapcomplete-current-open-changeset-cycle_infra" v="111520288"/>
<preference k="mapcomplete-current-open-changeset-cyclofix" v="97611926"/>
<preference k="mapcomplete-current-open-changeset-etymology" v="113127802"/>
<preference k="mapcomplete-current-open-changeset-trees" v="120616111"/>
<preference k="mapcomplete-hidden-theme-grb-enabled" v="true"/>
<preference k="mapcomplete-hidden-theme-httpsgistgithubusercontentcomriQQ4c129c9f2e700a971bd2f9374b86e29fraw-enabled" v="true"/>
<preference k="mapcomplete-hidden-theme-toerisme_vlaanderen-enabled" v="true"/>
<preference k="mapcomplete-installed-theme-amenitybicycleparking-combined-0" v="eyJpZCI6ImFtZW5pdHliaWN5Y2xlcGFya2luZ…idmVyc2l"/>
<preference k="mapcomplete-installed-theme-id-combined-3" v="CwiaWNvbiI6eyJyZW5kZXIiOiIuL2Fzc2V0…W5kbGlu"/>
<preference k="mapcomplete-installed-theme-id-combined-4" v="ZyI6MX1dLCJyb2FtaW5nUmVuZGVyaW5ncyI6W119"/>
<preference k="mapcomplete-installed-theme-id-combined-5" v="sIndpZHRoIjp7In…W119"/>
<preference k="mapcomplete-language" v="de"/>
<preference k="mapcomplete-layer-gps_track-enabled" v="true"/>
<preference k="mapcomplete-pictures-license" v="CC0"/>
<preference k="mapcomplete-unofficial-theme-berlin_emergency_water_pumps-combined-0" v="{&quot;id&quot;:&…ded&quot;}}"/>
<preference k="mapcomplete-unofficial-theme-berlin_emergency_water_pumps-combined-length" v="1"/>
<preference k="mapcomplete-unofficial-theme-bookcases-combined-9" v="e guardan libros&quot;}}"/>
<preference k="mapcomplete-unofficial-theme-httpsgistgithubusercontentcomriQQ4c129c9f2e700a971bd2f9374b86e29fraw-combined-0" v="{&quot;id&quot;:&qu…t;en&quot;"/>
<preference k="mapcomplete-unofficial-theme-httpsgi…&quot;}}"/>
<preference k="mapcomplete-unofficial-theme-httpsgistgithubusercontentcomriQQ4c129c9f2e700a971bd2f9374b86e29fraw-combined-length" v="2"/>
<preference k="mapcomplete-unofficial-theme-parking-combined-0" v="{&quot;id…t autoparkings&quot;}}"/>
<preference k="mapcomplete-unofficial-theme-parking-combined-length" v="1"/>
<preference k="mapcomplete-unofficial-theme-template-combined-0" v="{&quot;id&… bloc"/>
<preference k="mapcomplete-unofficial-theme-template-combined-1" v="ks), but make sure 'en' is everywhere&quot;},&quot;shortDescription&quot;:{&quot;en&quot;:&quot;The welcome message goes here&quot;}}"/>
<preference k="mapcomplete-unofficial-theme-template-combined-length" v="2"/>
<preference k="overpass-ide_query_0_0" v="p=2&amp;n=Trees without Ref&amp;q=LyoKVGhpc…r8WxMjVdOwovL8SPxJTEnXIgxZFzd"/>
<preference k="overpass-ide_query_0_1" v="Wx0cwooC…LHoXNrZcS9cXQ7"/>
<preference k="overpass-ide_query-count" v="1"/>
  </preferences>
</osm>

bryceco commented 2 years ago

Thanks for this!

bryceco commented 1 year ago

I think the way forward on this is to add import/export buttons like we did for quests.

tordans commented 1 year ago

I think the way forward on this is to add import/export buttons like we did for quests.

That would help for some use cases, yes. StreetComplete recently added a QR code feature, that allows to share configurations via URL/QR-Code. That is great, because it allows to start a mapping campaign by sharing a certain set of quests with users. (Which we did with @fixmyberlin).

The user preferences has a few advantages in special szenarios:

I can share my settings between ipad, macbook and iphone just by logging into the same account
I could modify "hack" the settings by editing the stored data; one could create an external editor for some cases with this
When a beta test makes me loose all my settings (which used to happen more often than now…), I just need to log in again

bryceco commented 1 year ago

Are you suggesting the QR code approach for quests or for custom presets?

bryceco commented 1 year ago

I implemented storing custom presets in user profile. Please give it a try (in 3.3.9)!

tordans commented 1 year ago

@bryceco are those custom presets synced between the GoMap MacOS App and the iOS App? I tried this today but it looks like this does not work(?)

bryceco commented 1 year ago

I would expect them to but maybe I'm wrong. Haven't tested it but I will when I get the chance (traveling so it will be a while).

bryceco / GoMap