Alpha 1

[lxbarth]

@ppawel -

What's your vision for a first iteration on the resurrection OWL API?

I know there are many great ideas floating around right now for improving OWL but there is also a pretty tough scalability problem to solve.

Based on what you see possible for a first iteration on the API I'd like to figure out what we expect from the owlviewer at that point.

/cc @samanpbb @zerebubuth @iandees

[ppawel]

Yeah, it got pretty wild with ideas :-) I think we should step back and focus on the core stuff for a while.

So - I've just finished an initial implementation of the tiler, it still needs (a lot of) testing, especially in the scalability department. Right now there is just a naive indexing scheme implemented - matching tiles by x, y and zoom.

On the data side currently the plan is to revisit the idea of importing planet next week when the new planet file is out.

I think for the first iteration we should also have some kind of simple aggregate tiles for low zoom levels. It should be easy to implement tiles with only the number of changes on it - this would at least allow the UI to function above ZL16 or whatever we choose as the "geometry" ZL.

[lxbarth]

I've been thinking about a first iteration some more.

I'd like to look at this work very much from what we (as OSM community) would like to see one day on openstreetmap.org and that's in my mind a question of evolution from the current changeset browser (history tab). This very evolution could be informed by a good OWL resurrection branch, so let me hash out what I'm seeing here.

The OSM.org changeset browser has many solid parts, but it suffers from three issues that diminish its every day usefulness:

1. Once in the changeset browser (=on the history tab), the currently viewed area can't be changed without leaving and reentering the browser.
2. For a given area, all changesets with a bounding box overlap are shown - even if they actually don't impact any data on the given area. This approach has a good reason - it's fast - but it leads to many irrelevant changesets polluting listings.
3. The interface does not highlight any modified geometry on the map.

(1) seems to be a straightforward UI level fix for openstreetmap-website.

(2) and (3) are the harder parts, and that's where current OWL work could be of benefit. I think we've done a great first step by breaking out an API that could power a future OSM.org changeset browser, but there are still tough scalability problems to solve and we're still working on defining UI paradigms and I feel this is where we need to sharpen our goals.

I'd like to suggest to take a very close cue from OSM.org's changeset browser and work very iterative.

Phase 1

• Fix (1). Fix UI on existing changeset browser - essentially update the changeset listing as the user zooms/pans on the map + clean up UI.
• Fix (2). Devise a blazing fast way to only list changesets that impact present area (this may be really hard and I'm still fuzzy on what the right solution would be here).

At this level we would have essentially what's today's OSM.org's history tab fixed. We would have a fast, responsive way of browsing changesets and we would only see those that truly matter. I think it's already hard work to get right here and it will likely involve plugging a specialized database (future OWL?) into OSM.org. But getting to this point will blow off the lid of the changeset browser and turn it into a tool that's actually useful. We could even do neat stuff like updating changeset listings as they come in creating a sense of movement.

Phase 2

• Fix (3). Display geometries impacted by a changeset. This is a tough problem. Our first tests with rendering geometry upfront are on the slow side - both for in-browser rendering and for delivering large geometries to the browser. Some changesets are very heavy (up to 50k edits) and we have no solid plan yet on how to scale to that. To keep the user interface clean and fast, I'm thinking that geometry would only be revealed after a user clicks on a changeset. But even if we do this, we still have to be able to handle a french village building import with 50,001 changes in some graceful way.
• Lower zoom levels: current OSM.org and the last OWL did not show any changes for lower zoom levels. For OWL there is now a prototype but I feel that for phase 1 we can either also punt on lower ZL's or just don't do anything different and show latest changesets also for low ZLs (this is right now possible on OSM.org btw, even if its not revealed in the UI).

@ppawel - I know you're doing all the heavy lifting on this work right now, so please read this note as first thoughts and recommendations. @samanpwbb and myself will be pretty much strapped of time for the upcoming months, but we would be interested to help push on the OSM.org UI improvements if backend pieces are coming together along the lines above or similar.

@zerebubuth @samanpwbb @ppawel @iandees

Looking forward to your thoughts.

[ppawel]

At this point the OWL backend and API is able to power most of the features you mentioned:

• It can serve tiles with changeset metadata and changeset geometry in GeoJSON.
• It can serve summary tiles for low zoom levels containing the number of changesets - this can be used to implement a "heat map" or a "marker cluster" functionality.

Owlviewer shows off these features nicely, see live instance:

http://suncobalt.dyndns.org:8082/owlviewer/index.html#3/57.33/41.57

What needs to be done is:

• Server-side performance and scalability testing

Basically - import planet, run replication, observe what happens (alternatively, import changesets "since forever" - could be done in parallel with a second database and feeding changes to the main OWL database). This will show how the current schema and tile indexing (which both are pretty naive, no magic involved) scale.

• Client-side performance and scalability testing

Based on what I've seen so far with owlviewer in different browsers, it may be a good idea to increase tile size for geometry tiles. Firing up a myriad of AJAX requests at every map change seems to kill browsers in subtle ways. On the backend side I can see tile requests being responded to in ~20ms each. On the browser side it takes about 2-3 seconds to download all tiles for the current viewport. When you look closer you will see that it's because those requests are spending most of the time in "blocking" state - waiting to be sent. Once they are sent, it's < 100ms to get the response.

Other problems that you mention are easier and do not affect architecture as much as the two above.

I'm planning to work on the server-side in the coming weeks. I was hoping to get some help with the client-side but I guess I'll have to plan for doing that as well now that you're focusing on other things.

One thing that would be great is if @samanpwbb could prepare some interaction/UI concept for integration with osm.org. Bits and pieces of owlviewer can be reused of course (especially when the Leaflet port hits osm.org) but still I am not competent enough to do the interaction design myself. This is not a time sensitive since in the coming weeks (maybe even months) I'll be working on scalability but it would be good if you could plan this into your TODO lists.

[ppawel]

FYI, I'm closing M1 (aka. Alpha 1). I think OWL works pretty well at this point as a proof-of-concept. In M2 I will be focusing on really testing scalability by importing more data.