search

nextcloud / maps

🌍🌏🌎 The whole world fits inside your cloud!
https://apps.nextcloud.com/apps/maps
GNU Affero General Public License v3.0
504 stars 89 forks source link

Switch to new tile provider #2

Closed jancborchardt closed 5 years ago

jancborchardt commented 8 years ago

Following the MapQuest open API shutdown we need to check for a new tile provider.

GNOME Maps uses Mapbox with a community API key over a GNOME proxy: https://mail.gnome.org/archives/commits-list/2016-July/msg07462.html I contacted them to find out how we can do it: https://twitter.com/jancborchardt/status/796054542144176131

A possible alternative would be the Wikimedia Cartographical project: https://www.mediawiki.org/wiki/Maps#Production_maps_cluster

cc @v1r0x @nextcloud/maps @schiessle

v1r0x commented 8 years ago

Definitely a problem. I remember that we switched from vanilla osm aka nominatim to something else for the routing/geocoder stuff, but is there a reason we don't use nominatim as tile server? Gnome's solution would be great too! Beside that, I'll look for another solution.

PS: I ordered a new laptop, thus I hope I'll have some more time for maps 😃

jancborchardt commented 8 years ago

@v1r0x yeah, sounds great! :) Also, @mattiasb of GNOME Maps replied to the tweets with further info, so we should contact Mapbox: https://twitter.com/jancborchardt/status/796054542144176131 (also cc @freenerd whom I know working there :)

(And cc @Dokhra @e-alfred @thomas-mc-work from the original issue cause we moved the repository here and will continue work here. :)

Henni commented 8 years ago

Mapbox would be great, but I haven't found any information on this "community API key" you mentioned @jancborchardt

jancborchardt commented 8 years ago

@Henni I’m pretty sure it’s something we need to mail them about. It sounds like mainly a courtesy & community support thing, not an official API key they offer. :)

e-alfred commented 8 years ago

Good to see the idea of a maps app isn't dead. Is Openstreetmap also of interest? Offline maps would be quite interesting as well to avoid the problem with an external provider cutting access suddenly.

Maybe this link could be of help:

https://wiki.openstreetmap.org/wiki/Offline_Openstreetmap

jancborchardt commented 8 years ago

@e-alfred while having a tile server or offline maps sounds good, it’s hardly doable:

Planet.osm contains the entire planet. This is a snapshot of the current data, usually from last Wednesday. This is almost 40 GB compressed.

Just Germany is 2.5 GB. ;)

Henni commented 8 years ago

@e-alfred still... Feel free to open an issue. It might be possible to implement something like offline area support, similar to others like Google Maps do it.

e-alfred commented 8 years ago

@Henni already did that https://github.com/nextcloud/maps/issues/12

Well, if it is that big, then there is probably no way around it (except for single countries for example).

DNkRockzzzzZ commented 7 years ago

I need this app running too.

Revertron commented 7 years ago

The settings of maps provide entering Mapzen API key, does it work?

v1r0x commented 7 years ago

The Mapzen API key in the settings is currently only for the routing machine.

jancborchardt commented 7 years ago

@Henni did you already mail Mapbox btw? :)

e-alfred commented 7 years ago

This bug report from Gnome seems to be a good reference point:

https://bugzilla.gnome.org/show_bug.cgi?id=764841 https://mail.gnome.org/archives/maps-list/2016-July/msg00017.html

e-alfred commented 7 years ago

Wikimedia is also working on a maps cluster, this would most probably be the best fit for an Open Source project like Nextcloud (Maps):

https://www.mediawiki.org/wiki/Maps#Production_maps_cluster

Henni commented 7 years ago

@e-alfred nice reference! The effort by wikimedia looks awesome. Main issue:

For now, maps.wikimedia.org only accepts requests from the domains in wmflabs.org, *.wikivoyage.org, or incubator.wikimedia.org.

So we might have to go with mapbox for now (similar to gnome).

And no I haven't contacted Mapbox yet. @v1r0x If you want you can do this. Otherwise I'll do that in a few weeks, when I'm done with my exams.

edit: wikimedia also has SVG support! :heart_eyes:

v1r0x commented 7 years ago

@Henni I'm currently very busy, so don't know when I have time for it. Should also work on the polls app and the maps-rework branch :sweat_smile: .

e-alfred commented 7 years ago

Maybe this map implementation for another selfhosted cloud service (Cozy Cloud) could be of interest:

https://github.com/RobyRemzy/cozy-map

jancborchardt commented 7 years ago

@v1r0x can you check out what they do and if we can use anything of it? :)

v1r0x commented 7 years ago

@jancborchardt pretty busy enjoying Scotland right now :wink: Back at home I should have some time for it. Or at the conference.

e-alfred commented 7 years ago

Is there any reason why not to use the same implementation of map providers like the awesome Gpxpod/Gpxedit/Gpxmotion apps?

https://gitlab.com/eneiluj/gpxmotion-oc https://gitlab.com/eneiluj/gpxedit-oc https://gitlab.com/eneiluj/gpxpod-oc

They implement Leafletjs to provide many map/tile providers using plugins. Many providers are available, as can be seen here:

http://leaflet-extras.github.io/leaflet-providers/preview/index.html https://github.com/leaflet-extras/leaflet-provider http://leafletjs.com/index.html

v1r0x commented 7 years ago

Leaflet is great, but it's annoying to work with plugins. They are either buggy or do not support the latest release. I work with leaflet at my job and it's frustrating. Useful plugins have many open PRs with bugfixes or features and nobody cares.

But I started to work on a complete rework of the app. As soon as I have more time for it, I evaluate what plugins we may want and how well they are supported.

e-alfred commented 7 years ago

@v1r0x I hope this great app can be brought back. A maps alternative where the user owns the data (not Google. Microsoft or whoever else) is something that is desperately missing from Nextcloud.

I also asked the GPXpod/edit/motion app developer if he is interested in helping out. Many features his apps provide would be very useful features if included in the maps app.

https://gitlab.com/eneiluj/gpxpod-oc/issues/77

jancborchardt commented 7 years ago

Would indeed really be cool if the folks of the other map-related gpx apps can collaborate here. :) cc @eneiluj

julien-nc commented 7 years ago

@jancborchardt Yes, why not ? We could first talk about where the map app is going.

Personally i'm not sure the "map" scope is small enough to fit in one single app. This would mean it could potentially include track collection management, track edition, track generation from routing engine, tile server hosting and every map-related thing. Maybe first thing to do is to determine a scope and even find a more explicit name.

A few questions that come to my mind :

I've got more questions that might be off topic. Should we open another issue to have a larger discussion ?

Here is the small chat with @e-alfred about map app.

v1r0x commented 7 years ago

Great to see you here @eneiluj! Since I wanted to rewrite the app anyway, it is a good point to discuss the scope. In my opinion it should be an alternative to stuff people would normally do with osm.org or gmaps + some additional features (contacts, ...). But maybe we should switch to another issue or chat @jancborchardt ?

Would be great to have another dev working on the app!

jancborchardt commented 7 years ago

Awesome @eneiluj, welcome! :) As @v1r0x already put it, the core scope is to provide an alternative to things like Google Maps, Apple Maps etc. This includes:

Basic core features:

Advanced stuff:

And that's just the core stuff that I can think of off the top of my head. :) It's already plenty. At the core Maps is, as most Nextcloud apps, an app focused on the majority of users as a viable alternative to proprietary platforms.

julien-nc commented 7 years ago

@jancborchardt thanks !

Indeed, it would be great to sync OSMand/Maps.me favourites with NC. As for the other core features you mention, i don't get why they should be available in a "personal" cloud. They are not very personal, the app would not store/read any personal data. Many public services (graphhopper, openrouteservice, osm...) already exist as good alternatives to gmap. Some of them can even be deployed in a local network.

I think a general "map" app should focus on everything related to personal data, for example :

What do you all think about that ?

I don't think it would be good to integrate complicated interfaces like the GpxPod one in the map app. It should stay clear and simple as it's going to mix many features. I don't remember what is possible in map app and how it was presented. Is it similar to OSMand "map configuration" menu ? This menu seems fine to me. Toggle things apparition (like contacts or track files) and go to sub menus to select elements, see more details.

Should we continue this discussion on help.nextcloud.com ?

jancborchardt commented 7 years ago

@eneiluj searches for places, setting favorites and routing are very personal things. Google Maps knows where you live, where you work, where you go, which places you like, where your friends live, where you are right now, where you just went to a restaurant etc.

That’s the real-world privacy aspect. They don’t only know which websites you visit, they also know your physical whereabouts and preferences. This is as much of a privacy concern as your other digital data, if not more. :)

Many public services (graphhopper, openrouteservice, osm...) already exist as good alternatives to gmap.

None of these are really well integrated with your other personal data like contacts, personal address etc. But we do have all that with the other Nextcloud apps. And that integration is key, and why Nextcloud is so important. It provides an integrated experience just like with Google Apps (where you have Google Maps) or the Apple ecosystem (where you have Apple Maps). :)

julien-nc commented 7 years ago

@jancborchardt As i said, i agree on favourites (and all personal data) but i suppose integrated search for places will be done with a plugin which uses an external service anyway (nominatim for example), because the map data is stored elsewhere. Problem is just moved from gmaps to nominatim.

To me, the problem is about which service we use. A private one which sells our private information or a more ethic one which goal is just to provide a service like OSM. I have no problem with using OSM for routing and if i would have, i could still use Tor to access it. Without using Tor-like system, there is no difference (in terms of privacy) between searching on openstreetmap.org and using a map plugin inside Nextcloud. In both cases, a request is done by my browser to nominatim.

What i meant was that routing and searching for public places (cities, buildings, addresses...) are not user-specific services. The service is the same for anyone who uses it. When i'm connected to my Nextcloud account, i'm exploring my personal space. I still don't see why there should be a gateway for external services in a Nextcloud app. Maybe i'm wrong but i think that is not what users look for in Nextcloud experience.

I agree that searching in the map is useful and should be done. I agree that routing can be useful IF a route can be saved directly in Nextcloud files. Considering your arguments about integration, those are interesting features, but i think it is not increasing user's privacy.

e-alfred commented 7 years ago

@eneiluj How much information would the tile provider actually get? They know which countries/towns/places I requested (depending on the level of detail) at a certain time, but do they have much more? Do they know everything else Google-style too? Contacts, calendar, pictures, routes and so on stay on the Nextcloud server.

Regarding a tile server and availability and privacy issues with them, would it be possible to download certain countries that the user is interested in or uses most locally and use them without any outside provider? Like it is done with offline navigation apps for mobile devices? I created an issue already here:

https://github.com/nextcloud/maps/issues/12

v1r0x commented 7 years ago

I agree with @jancborchardt. I also think it's a good idea to integrate search, routing, etc even if they are not user-specific services. An all in one solution is far more better than checking your favorites or contacts in nextcloud and then switching to osm in another tab (or open a new one and load all the js stuff). So you have all maps-related stuff in one place.

julien-nc commented 7 years ago

@e-alfred If you use openstreetmap.org for searching, your contacts also stay on your Nextcloud instance :smile: If you integrate the search engine to Nextcloud, your request still go to nominatim... The only difference is the potential integration with other Nextcloud apps/data.

I like the idea of providing downloaded maps. It could be done with a sort of tile server hosted on Nextcloud server. Tiles archives grouped by area could be generated and hosted by app maintainers and provided to NC admins.

@v1r0x I generally prefer to find one tool per task/usage to be able to select the best one for each case. One big criteria to choose is, of course, interoperability so that all my tools can be combined. I work hard :wink: to understand the fact that most people prefer all-in-one solutions, even if they would like to replace a sub-part of it or if makes it harder to separate the data and the tool.

Switching between tools is not a problem to me. It is even a good sign. It means there is a good diversity of tools, i can choose my favourites and, as there are standard file formats, i can combine tools that don't know each other.

I think loading another web page and switching between tabs should not be considered as painful or excessive. Yes it sounds appealing to be able to see the weather forecasts integrated in the map i use to plan a ride. But i think tools are more useful if they are simple and atomic. I also want to choose my forecast service. Let's avoid the google/apple syndrom :wink:

Maybe it's a question of granularity... I see Nextcloud as "the tool" or "the all-in-one tool". Inside Nextcloud, i like to be able to choose between "mail" and "rainloop" apps as my web mail client. Both of them can use information stored by the Contact app. Creating an all-in-one map app would mean fixing too many features together like if "video calls" and "contacts" were joined in a single app. As far as i know, no app combines a lot of features. Even in terms of interface, it would be hard to conceptualize, to implement and to use.

Developing three apps around "map" topic does not prevent me to link them but they can still be used as small independent tools. Maybe my way to contribute here is (except from writing long issue comments), for example, to provide a gpx editor which can be accessed easily inside a Nextcloud instance from another app. In "map" app, when a track is displayed, there could be an "edit" button which just loads this track in GpxEdit.

Gee, isn't there a size limit to comments ?

e-alfred commented 7 years ago

The old Unix philosophy at work: "Do one thing, do it right." A lot of apps doing different things aren't probably a bad idea, also from a maintenance point. If one breaks for whatever reason, then the others are still useful (given they are not facing an identical problem).

Regarding a sefhosted tile renderer, there is a project called Openmaptiles that provides a docker container serving a tile renderer and downloads of tile data for all regions separately:

https://openmaptiles.org/docs/host/tileserver-gl/ https://tileserver.readthedocs.io/en/latest/

This would make a similar setup analogous to Collabora CODE or Onlyoffice which have a Nextcloud app that connects to a selfhosted service (mostly running on docker which is the easiest to set up) for editing office documents.

They also provide a tile renderer as an NPM version and even a PHP version:

https://openmaptiles.org/docs/host/tileserver-php/

jancborchardt commented 7 years ago

Search and routing are very much user specific as they include recent searches, often searched places, favorite routes, and can automatically get from co tacts where you work and live. For example the route start can be set to "Home" with no problem since your Nextcloud has that data, as it does the addresses of your friends. OSM does not.

julien-nc commented 7 years ago

@jancborchardt i agree, the integration aspect is interesting.

What do you think about this one ? Let's imagine a gpx track is loaded and displayed in maps app. Before anyone develops a gpx editor in maps app, if GpxEdit is installed on the Nextcloud instance, there could be an "edit" button that opens a tab with this track loaded in GpxEdit. Just like it's possible to do in the Files app "three dots" menu. Same for animating with GpxMotion. Same for comparing with GpxPod comparison page.

jancborchardt commented 7 years ago

@eneiluj yes, that would be a good first step of integration! :) The future ideal would be that it's all one app and a seamless flow - of course we have to move in small manageable steps.

e-alfred commented 7 years ago

@jancborchardt Do you have some kind of roadmap what cloud/should be tackled first?

Spartachetto commented 7 years ago

I have the impression that several different issues are mixed in this thread

Tile provider

If do not want to deal with any problem you can use a tile provider (basically you request the tiles to a website). You could use openstreetmap (which provides different tilesets), Mapbox or others.

Issues

Libraries to use tile providers

Leaflet is just a library that allows to use different tile providers...

Tile server

If do not want to have any limitation you can use your own tile server. There are opensource ones which use free data (again, think to openstreetmap). You can personalize the information that you display in your maps (Like having red streets or blue hospitals, or displaying prominently info about e.g.: electric grid)

Issues

Geolocation

It means "the service that takes Strasse des 17. Juni 136, 10623 Berlin and gives back 52.5139707, 13.3254422". It is simpler than tiles! Here you have the same issue of the tiles:

routing

"How to reach B from A". Complex from a mathematical point of view. Do you want car, pedestrian, bike? Do you want the shortest path, the safest, the quickest, .... Same issue again:

Sorry for the long text, hope this helps!

jancborchardt commented 7 years ago

@eneiluj regarding roadmap what could/should be tackled first, see my comment above: https://github.com/nextcloud/maps/issues/2#issuecomment-313360509 - the items are listed in order of importance. :)

jancborchardt commented 7 years ago

Also, we should first start out with simply using 3rdparty services like OSM and graphhopper for routing - like the Maps app did previously. Hosting own tile provider and routing server is out of scope for many shared hosters

e-alfred commented 7 years ago

Here is also a pinning app for Owncloud that integrated maps too:

https://github.com/libasys/pinit

jancborchardt commented 7 years ago

FYI that app seems unmaintained. :)

We should simply also use leaflet with osm for the Maps app, then we can continue with the rest. :)

e-alfred commented 7 years ago

I know, but it still has some interesting pinning functions that could be of interest in a future maps app.

For now, leaflet.js seems to be the best way to get the maps app working again. Basic functionality trumps everything else. The map types used by GpxPod are a pretty good choice I think and work pretty well.

v1r0x commented 7 years ago

I started to work on the rework branch again. We decided to start with tiles directly from osm and (try to) collaborate with Gnome, MapBox, etc. when we hit osm restrictions or any other limitations.

e-alfred commented 5 years ago

I think this bug report can be closed now? The new implementation works fine and should do for some time (unless OSM goes down or leaflet.js gets unmaintained which seems not that likely at the moment).

Maybe also some other unneeded bug reports and pull request can be closed now too?

jancborchardt commented 5 years ago

Yes, with the new implementation this can be closed, thanks for noting @e-alfred! :)