search

osmlab / editor-layer-index

A unified layer index for OSM editors.
https://osmlab.github.io/editor-layer-index/
Other
220 stars 258 forks source link

ELI's future #586

Open grischard opened 5 years ago

grischard commented 5 years ago

I've been thinking about where to take this project next for a while. The current index, which was a perfect solution when we had a dozen layers, is really showing its age.

The solution should use off-the-shelf when possible. This means existing software, existing APIs, code that already works.

The solution should let the clients query about any kind of local features, whether they're imagery, communities, or anything locally relevant we can think of (tagging presets? Validation rules?).

There have been a few similar solutions or attempts: the osm Imagery Offset Database, https://github.com/osmlab/osm-community-index/issues/80, maybe the JOSM chat plugin, geocoders if you really stretch the definition.

WFS is possible, but is slow when not backed by elasticsearch and antiquated always. Parsing XML is a pain. Something like python FeatureServer (which seems dead) or JavaScript FeatureServer (which seems immature) would maybe work. Maybe elasticsearch's native geospatial support can do the trick. I really haven't played with any of these yet.

The data could still be integrated from ELI at first, but ideally I'd like something a bit more newbie-friendly to be made available, with a form you can fill out.

What do you think?

simonpoole commented 5 years ago

Given that've suggested similar things I suppose I should comment.

I've been thinking about where to take this project next for a while. The current index, which was a perfect solution when we had a dozen layers, is really showing its age.

* Everyone downloads 1.75 MB of geojson, most of which is in areas they're never going to edit

* Everyone downloads all entries, even if our clients don't support the wms or wmts catalog endpoints.

We only have three editors P1, P2 and iD for which we can assume that network connectivity is present, essentially everything else will as a tendency pre-download/bundle/whatever the data and this should still be possible post any changes that are made.

Obviously any such system should allow generating a configuration that only contains supported sources (I doubt that anybody ever included WMS sources if they were not supported, they were simply removed before packaging).

* It's impossible to specify that a layer is "best" in some places but not others.

* It's impossible to store extra local attributes like imagery offsets

Both imagery offsets and quality assessments would seem to share a couple of similar properties that might make it possible to include them in one system (as noted above there should be a facility to store this locally). 

* The index only stores imagery. Local communities are another, incompatible project. For things like address display rules, local language preferences or local defaults, we don't even have a centralised index, and creating one would be hard.

This would seem to be substantially off topic and out of scope. 

* The [JOSM schism](https://josm.openstreetmap.de/wiki/ImageryCompare) is wasting volunteer time.

* The output is statically included in iD and only gets updated when a new version gets released.

* Contributing a new layer can be confusing for newbies.

The solution should use off-the-shelf when possible. This means existing software, existing APIs, code that already works.

I've already suggested that OAM might be a suitable starting point for a DB based index, I however don't think that this will work without additional work. The other issue is that OAM is likely in its stasis between rewrites to justify obtaining new funding for it, it is not quite clear to me how we could move OAM out of this loop so that we could have a multi-year (lets say around 10) stable base to work from and improve on.

The solution should let the clients query about any kind of local features, whether they're imagery, communities, or anything locally relevant we can think of (tagging presets? Validation rules?).

There have been a few similar solutions or attempts: the osm Imagery Offset Database, osmlab/osm-community-index#80, maybe the JOSM chat plugin, geocoders if you really stretch the definition.

WFS is possible, but is slow when not backed by elasticsearch and antiquated always. Parsing XML is a pain. Something like python FeatureServer (which seems dead) or JavaScript FeatureServer (which seems immature) would maybe work. Maybe elasticsearch's native geospatial support can do the trick. I really haven't played with any of these yet.

The data could still be integrated from ELI at first, but ideally I'd like something a bit more newbie-friendly to be made available, with a form you can fill out.

What do you think?

I think we should avoid feature creep :-)

grischard commented 5 years ago

for which we can assume that network connectivity is present

If you're accessing imagery, yes, I'm assuming network connectivity is present :). Information can also be cached for quite a while.

I've already suggested that OAM might be a suitable starting point for a DB based index

I see OAM as one of the possible sources. Whatever its metadata comes as, we could integrate it.

imagico commented 5 years ago

Both imagery offsets and quality assessments would seem to share a couple of similar properties that might make it possible to include them in one system (as noted above there should be a facility to store this locally).

One important thing to consider is that for those it would be paramount to consider what mechanisms are used to provide and to update such information. Right now all the image layer metadata here is hand maintained - which to a significant extent defines the value of this index. Broadly including image quality data however would not be feasible to do by hand. This would either be crowd sourced data (like for offsets) or data supplied by the image providers. For the former this depends on editor integration, for the latter it would be very significant to create well usable mechanisms so the hurdles are low to generate such data in a compatible form which would massively increase the chances that image provider actually do so.

simonpoole commented 5 years ago

for which we can assume that network connectivity is present

If you're accessing imagery, yes, I'm assuming network connectivity is present :). Information can also be cached for quite a while.

At least JOSM and Vespucci will cache imagery tiles locally so no network connectivity needed (you obviously one way or the other still need the meta information for the imagery in question).

Marc-marc-marc commented 5 years ago

The current index, which was a perfect solution when we had a dozen layers, is really showing its age.

I find that there are many points very different from each other:

I think it is necessary to solve "internal" (= related to imagery it-self) problems before talking about the problem of compatibility with other related projects (community, offset,...)

* Everyone downloads all entries, even if our clients don't support the wms or wmts catalog endpoints.

this look like easy to fix (make may produce one file per type for ex)

* Contributing a new layer can be confusing for newbies.

a web-form would be useful. reusing one of the many layer testers would be convenient. a tool to test/get the real geographical scope and simplifies it to define the layer scope would be a real pleasure.

What do you think?

split all issues into several "one-at-a-time" & "as-small-as-needed" issues each one easy to resolve. and start with adding a schema version :)

don-vip commented 5 years ago

@grischard it's not OK to talk about a "JOSM schism". @stoecker created the concept of central imagery sources repository in october 2010 and it was open to the OSM community from the very beginning.

It's Ian who initiated the schism when he started EII in march 2013.

You're absolutely right that the schism created by EII/ELI is a waste of effort. But please don't rephrase history to make us look as the bad guys.

If you have doubts about the future of this project can you please consider the obvious solution: drop it and reuse our system, like Vespucci does with our presets? XML is not a problem as we already serve geojson.

If any technical reason makes this transition not possible right now, please let us know and we can work on that.

iandees commented 5 years ago

It's really nice waking up in the morning to an attack in the OSM community. 🙄

The goal behind Editor Imagery Index was to create a more friendly experience all around. For people who want to submit new imagery layers, the GitHub ticket interface is nicer than Trac's ticket interface or editing an XML file in a tiny Trac wiki editor. For people who maintain the list, pull requests and the continuous integration system is easier and more robust method of adding to the system. ELI is still better in these important areas.

As has been mentioned up thread, we should file tickets to discuss the specific problems brought up in this thread and work on/discuss those problems specifically.

simonpoole commented 5 years ago

There are two sides that are both a bit right and a bit wrong wrt the schism aspect of ELI and I can understand both sides of the argument a bit, not the least because I'm using ELI on the one hand and originally JOSM derived presets on the other hand in Vespucci.

When I chose to use ELI back in 2014 or so, the JOSM imagery list format had gone completely stale and it clearly was not going to see any love any time soon. That it is now competitive again has 100% to do with the fact there is competition in the form of ELI. So I can't fault @iandees for doing something different, in particular I essentially every day have to live with the downsides of extending a format were the maintainer is 100% resistant to improvements see http://vespucci.io/tutorials/presets/#extensions (there are more in the upcoming release).

Now as I've pointed out multiple times the current solutions to maintaining the imagery lists are really not far away from collapsing under their own weight, so instead of fighting about who is winning now, could we perhaps simply work towards an unified system to replace both?

grischard commented 5 years ago

Sorry for the ruffled feathers. The schism happened way before I even knew about this project, and I did not want to blame anyone. Historical reasons or how it happened, honestly, never really interested me - I've always wanted to close the gap. You know I'm primarily a JOSM user.

I have updated the phrasing in my ticket. I think, in the end, we're all on the same side on this.

andrewharvey commented 5 years ago

If you have doubts about the future of this project can you please consider the obvious solution: drop it and reuse our system, like Vespucci does with our presets? XML is not a problem as we already serve geojson.

@iandees you're spot on about ELI on GitHub being more friendly. I favour how ELI uses GitHub, including Issues and Pull Requests which are essential for maintaining the imagery index. Using the JOSM wiki as an imagery index lacks automated CI/CD, issues, and a temporary staging ground for edits before they go live.

andrewharvey commented 4 years ago

See also https://github.com/openstreetmap/iD/issues/4994#issuecomment-582418839 hint's that iD might move away from the editor-layer-index in the future, but it's too hard to tell what their plans are at this stage.

bhousel commented 4 years ago

See also openstreetmap/iD#4994 (comment) hint's that iD might move away from the editor-layer-index in the future, but it's too hard to tell what their plans are at this stage.

I'm switching iD from editor-layer-index to the replacement that I built. Details here: https://github.com/openstreetmap/iD/issues/7425

Please check it out and follow us over to: https://github.com/ideditor/imagery-index Thanks for your years of service, editor-layer-index! @grischard, feel free to kill it off 👍

simonpoole commented 4 years ago

No I would suggest banning bhousel from any OSM related dev for life.

andrewharvey commented 4 years ago

In the interest to try and work out a way forward, as far as I know those using osmlab/editor-layer-index include:

I'm trying to get a feel from all the known users of this project to determine if we should continue to maintain osmlab/editor-layer-index in parallel to ideditor/imagery-index or if we'll be able to just maintain one of these index projects.

bryceco commented 4 years ago

I don’t plan to switch but I just now became aware of iD’s version. If iD’s version would reduce Go Maps!!’s network usage that would be a win, but I feel like ELI is more stable for now (not knowing whether iD will make breaking changes without due notification to other consumers). I wouldn’t consider JOSM’s list because 1) it would require writing more code to switch over and 2) I prefer an Apache style license over GPL.

simonpoole commented 4 years ago

@andrewharvey the value of this repo has mainly been created by the large community of OSM contributors that have researched and provided the contents, originally to the JOSM wiki and since a couple of years here and a similar observations goes for the NSI and the community index.

The maintainers have a fiduciary obligation towards that community to maintain its contents in stable and accessible fashion that can be utilized by as many applications as possible.

With a dev hat on: I'm definitely not against technical change in how we manage the contents and what it contains (see this thread and its predecessors), but just as @bryceco says, this needs to happen in a predictable and collaborative way, that allows migration in an organised fashion and does not depend on the whims of an individual.

And while that surely goes for any kind of project, it is in particular of importance for volunteer run undertakings, we neither get more time to burn or money from unnecessary churning. Not to mention that for non-web apps release cycles tend to be far longer and we need to maintain support for older versions for a reasonable time as we can't simply replace them on user devices by snipping our fingers.

stoecker commented 4 years ago

I wouldn’t consider JOSM’s list because 1) it would require writing more code to switch over and 2) I prefer an Apache style license over GPL.

You should properly inform yourself first to prevent such misconceptions. JOSM offers data in GeoJSON format as well and license is CC-BY-SA (and parts also LGPL) for the maps list.

I feel like ELI is more stable for now

Note that JOSM has more than 500 unfixed ELI issues in our compare list. I wouldn't call this stable: https://josm.openstreetmap.de/wiki/ImageryCompare

systemed commented 4 years ago

Potlatch 2 uses this index, @systemed are you planning on continuing to use osmlab/editor-layer-index

I wasn't planning to switch away but then I wasn't previously aware of this latest episode of hobbydrama. When you kids have finished beating each other up in the sandpit I'll just choose the most easily parsed survivor.

grischard commented 4 years ago

No name calling in my issue tracker! If you do, I will introduce subtle hard-to-debug bugs in the editors you maintain :-p.

What the fork means is that we will have a lot less contributions here, and that this index will inevitably become outdated and die.

I recommend for now that Vespucci and GoMap!! switch to https://josm.openstreetmap.de/maps?format=geojson which is plug-and-play compatible. Gzipped, it's 651.60 KiB.

The license of both databases is CC-BY-SA; see https://github.com/osmlab/editor-layer-index/blob/gh-pages/LICENSE and the very bottom, very right of https://josm.openstreetmap.de/wiki/Maps

It would be nice if our friends at JOSM could provide an imagery.xml for Potlatch by running https://github.com/osmlab/editor-layer-index/blob/gh-pages/scripts/convert_geojson_to_legacyjson.py on their side.

stoecker commented 4 years ago

It would be nice if our friends at JOSM could provide an imagery.xml for Potlatch by running https://github.com/osmlab/editor-layer-index/blob/gh-pages/scripts/convert_geojson_to_legacyjson.py on their side.

We will not call such a script, but we can adapt our export functionality to the desired format when there is real demand for it. As said years ago: If someone wants to use JOSMs map database and technical issues prevent this, simply open a ticket with the what and why and we'll find a solution.

simonpoole commented 4 years ago

I recommend for now that Vespucci and GoMap!! switch to https://josm.openstreetmap.de/maps?format=geojson which is plug-and-play compatible. Gzipped, it's 651.60 KiB.

Unluckily while that is correct in that the schema is the same, it doesn't contain the same information, as you know. I've already mentioned the other practical issues above, and additionally we need to change documentation to redirect contributors to the JOSM website for now.

@frodrigo would you consider replacing the imagery configuration in the iD instance you are running with the data from JOSM? In particular with an eye to replacing the instance of iD available from osm.org.

Marc-marc-marc commented 4 years ago

it is important to realize that 8 contributors alone make half of the commits (not to mention that among these, there are many large commits). Among these 8 contributors, several were working on data reconciliation/synchronization between eli and josm (both on the schema and the data). What are they going to do? Does it make any sense to have to rewrite all the comparison/synchronization tools eli<>josm just to follow the new fork? the question arises. so whether or not this repository dies is not the right question. the right question is: what is best for the community? in the same way that we don't ask an osm contributor to send his changeset several times to several database, once per editor, we should succeed in not asking an osm contributor to send his layer contribution several times.

simonpoole commented 4 years ago

the right question is: what is best for the community?

I think that is a rather rhetorical question, naturally it would be best to just have one place to submit and have imagery sources vetted.

However that would need cooperation from all parties (and the historical lack of that is the reason I now and then point to the fact that starting off this repo was not just for gratuitous reasons), that means accommodating data, functionality and workflows that ones own project does not require.

A likely corollary of that is that wherever this information is collected, it would best be at least semi-neutral ground and be run as a separate project.

grischard commented 4 years ago

@stoecker Simon is talking about the privacy policy URL attributes. JOSM currently already has the eli-best attribute, which it doesn't use itself. Would it please be possible to add the privacy policy URLs to the JOSM index, even if you don't use it, so that Vespucci can?

This would be a loud, visible, positive step showing that the JOSM developers care about other editors using their index. Please :).

Klumbumbus commented 4 years ago

I hope you (@bhousel) are aware that maintaining such an layer index is a ton of work and more crucial: constant work. 3 indexes are too much and don't help at all in terms of content quality, contributors experience and users experience. Even 2 indexes were too much. (I personally won't contribute to ideditor/imagery-index.)

don-vip commented 4 years ago

@stoecker Simon is talking about the privacy policy URL attributes. JOSM currently already has the eli-best attribute, which it doesn't use itself. Would it please be possible to add the privacy policy URLs to the JOSM index, even if you don't use it, so that Vespucci can?

This would be a loud, visible, positive step showing that the JOSM developers care about other editors using their index. Please :).

I've reopened https://josm.openstreetmap.de/ticket/17285

simonpoole commented 4 years ago

@Klumbumbus I suspect the plan is to draw all volunteer resources to his website and repo, likely by integrating the messaging around it in to iD and leveraging the more or less automatic deployment on osm.org to redirect people.

Klumbumbus commented 4 years ago

https://github.com/osmlab/editor-layer-index/blob/gh-pages/README.md#using-this-index should be updated. Which editor uses this index now?

andrewharvey commented 4 years ago

https://github.com/osmlab/editor-layer-index/blob/gh-pages/README.md#using-this-index should be updated. Which editor uses this index now?

According to https://github.com/openstreetmap/iD/pull/7428#issuecomment-601810337 iD is using, JOSM is still optional, have any of the other ones changed?

By the way I was reading https://josm.openstreetmap.de/wiki/Maps today and surprised to see it say " is pretty much dead since the iD editor started to use an ​own list." in light of https://github.com/openstreetmap/iD/pull/7428#issuecomment-601810337 I'm not sure if that still applies. I can understand it's hard to keep on top of this though with all the back and forth changes.

I got a bit de-motivated by the whole situation and stopped looking here, but just recently started to try and help with maintenance given iD may actually still be using it.

andrewharvey commented 4 years ago

Looks like GoMap!! uses JOSM's index now -> https://github.com/bryceco/GoMap/blob/master/src/Shared/AerialList.m#L712

Klumbumbus commented 4 years ago

OK, I didn't know that iD switched back. (I changed the comment in the JOSM wiki.)

GoMap switched to the JOSM list, see above my last comment. Not sure about Vespucci.

andrewharvey commented 4 years ago

Yeah seems Vespucci also moved to JOSM https://github.com/MarcusWolschon/osmeditor4android/commit/552cee76538dc13266e27d0885ba37516b5e1f97

andrewharvey commented 4 years ago

For what it's worth @Klumbumbus I still think that using a system that allows changes to be proposed and reviewed with a place for comments is a good idea, and one of the advantages of something like GitHub or GitLab.

A new contributor can make a first draft pass. Maintainers can check it looks good and passes licensing checks before it is accepted.

A place for issues is nice to have to track sources that can't yet be added, but as a place to discuss adding it to the index in the future.

simonpoole commented 4 years ago

To be clear: the current Vespucci release (14.1) uses ELI, the upcoming release uses the JOSM configuration, however users can still update/replace from ELI if they so wish.

Both systems (ELI and JOSM) have numerous UI and scaling issues and should be replaced (probably by an API) with something that also manages other types of third party data, and requires review before a specific addition goes live.

bryceco commented 4 years ago

Go Map!! Is using JOSM but I have no loyalty. There have been configuration problems on both so I’ll go with whichever is most stable.

tordans commented 3 years ago

For Reference: https://github.com/openstreetmap/iD/issues/8086