[SIP-63] Incorporating other map sources in Superset

gk1089 commented 4 years ago

[SIP] Proposal for Incorporating other map sources in Superset

Motivation

Superset currently uses MapBox to display maps which has its own limitations. There are other map sources available as Web Map Services (WMS). For example, Open Street Maps (OSM) that can be called upon using many js libraries like Leaflet or OpenLayers. This would pave way for more GIS functionalities in SuperSet in true open source spirit.

Later, the same could also apply to the deck,gl visualizations that simply use mapbox for its simplicity.

Proposed Change

Module can be created that permit adding WMS services during the creation of charts. Other features may include:

Adding multiple layers and providing facility to turn them on/off.
Permitting clicking on features (points) to show a popup with more data/links on the map itself.

New or Changed Public Interfaces

Same as described above.

New dependencies

Leaflet (https://www.npmjs.com/package/leaflet) and OpenLayers (https://www.npmjs.com/package/ol)

Migration Plan and Compatibility

Should work with existing setup.

Rejected Alternatives

I am proposing considering the open data sources right now. So, no Google/Bing maps etc.

Describe alternative approaches that were considered and rejected.

I would like to attempt this but I am really at a loss for where to begin with implementing it. I have seen discussions about new chart types here and although the documentation is old regarding that, some of the issue threads provide pointers in the right direction.

I intend this thread to be something of the same kind where some of the developers may point us to the right section of the code where we can begin exploring and implementing open source mapping visualizations.

I hope the developers see some merit in it.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

dharamgajera commented 4 years ago

Any Update on this ?

juen1jp commented 4 years ago

I would be interested in engaging with the community about this. MapBox is a non-starter for us and a solid block for us to use Superset. Are there plans to implement support for open mapping projects? We would be interested in helping to get open mapping projects added so the package is a truly open source solution.

villebro commented 4 years ago

@juen1jp I'm sure a contribution would be warmly welcomed by the community. As it will probably be a major change, I would propose opening a SIP (Superset Improvement Proposal) to make sure the community is able to chime with ideas and concerns prior to any major development work taking place.

gk1089 commented 4 years ago

@juen1jp I'm sure a contribution would be warmly welcomed by the community. As it will probably be a major change, I would propose opening a SIP (Superset Improvement Proposal) to make sure the community is able to chime with ideas and concerns prior to any major development work taking place.

@villebro I opened this thread as a SIP itself. Are you proposing a fresh SIP be opened?

I just need a few indicators regarding what part of the code to look into, as in, the code files that make the mapbox magic happen. Just searching with 'mapbox' keyword in the code files has not given me much to go upon.

villebro commented 4 years ago

@gk1089 my bad, I didn't notice this was in fact opened as a SIP as it wasn't indicated in the title. The title should be prefixed with [SIP-XX] where XX is the next available unused SIP id. I believe SIP-50 is the next available id.

I assume most mapping systems require some for of token. This should be added to config.py so that it can be defined in the local superset_config.py. Currently the mapbox token is returned by viz.py to the relevant visualizations; I believe the plan is to return this type of data in the bootstrap payload in the future. The committers can surely help with the details once we get that far. I propose reaching out on Slack, e.g. the #visualization_plugins channel.

gk1089 commented 4 years ago

@villebro Thanks for your response. As far as I understand it, tokens are compulsory only for metered map services like MapBox, ESRI, Bing, Google map services etc. For the open map services like Open Street Map (OSM), this is not a requirement and maps can be accessed directly by including the corresponding (Web Map Service) WMS URL through some js libraries like Leaflet or OpenLayers.

For example, I found the WMS URL (http://maps.heigit.org/osm-wms/service?REQUEST=GetCapabilities&SERVICE=WMS) for a world map on the info section of https://www.osm-wms.de/, which in turn was found on the OSM page on WMS (https://wiki.openstreetmap.org/wiki/WMS). I can use this WMS URL to show a map in my own page and the corresponding Leaflet/OpenLayers library will provide me tools to navigate on the map, show popups, make measurements etc.

It would be great to have the same control through a custom Superset visualization.

I shall try to join the slack and discuss it there and I request @juen1jp to chime in with his ideas. Meanwhile, may I request you to tag contributors who may have worked on the mapbox module in the past to share their take on this in this thread.

gk1089 commented 4 years ago

@villebro May I also request for this thread to be marked as active. It is currently marked as 'closed' by the activity bot.

villebro commented 4 years ago

In case there is no need for a token, then this will be quite straight forward, and should not be any different from any other new viz plugin. Please see the Hello World blog post on how to get started of you haven't already.

gk1089 commented 4 years ago

In case there is no need for a token, then this will be quite straight forward, and should not be any different from any other new viz plugin. Please see the Hello World blog post on how to get started of you haven't already.

Thanks! I was not aware of this post.

While I have a go at it, for the sake of reusing the code is it possible to point to the files which deal with the existing MapBox code?

roaur commented 4 years ago

I was curious to know the status of this SIP so I was digging around in the repo and saw that SIP-50 already exists in the SIP Project board. Here's SIP-50 as it exists on the project board: https://github.com/apache/incubator-superset/issues/10418

May I suggest this SIP number be updated to 53?

mistercrunch commented 4 years ago

Most plugins live here: https://github.com/apache-superset/superset-ui/tree/master/plugins

DeckGl plugins here: https://github.com/apache-superset/superset-ui-plugins-deckgl

mistercrunch commented 4 years ago

We recently made the resolution to use ECharts as our primary visualization library https://github.com/apache/incubator-superset/issues/10418

Do the maps in ECharts depend on mapbox?

junlincc commented 4 years ago

https://echarts.apache.org/en/download-extension.html I believe mapbox is one of them..

On Fri, Aug 21, 2020 at 3:54 PM Maxime Beauchemin notifications@github.com wrote:

We recently made the resolution to use ECharts as our primary visualization library #10418 https://github.com/apache/incubator-superset/issues/10418

Do the maps in ECharts depend on mapbox?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-superset/issues/9281#issuecomment-678547963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQFR5UYRTCOKNNUYCUMVGFDSB33KRANCNFSM4LFTS4TA .

junlincc commented 4 years ago

🏷chart

juen1jp commented 4 years ago

I think there is some confusion. Those of us who work for corporations don't want a commercial provider (Mapbox, Google, etc)

We need to keep to freely available solutions. This generally looks like running our own tileset servers and then something like leaflet to serve the mapping visualization.

ECharts looks nice and even has a rudimentary leaflet plugin. My only initial concern is that it is introducing more layers of complexity; however, that may be fine.

In general, I am new to the community. I am currently scoping the work that would need to be done in order for us to use this project. Defining a good default visualization framework is great. Outlining a formal way to contribute plugins and defining how to create a plugin is much better.

It looks like echarts will make an excellent default for the visualizations. However, realistically we will probably have to create some more custom working visualizations for our use cases. If they are generalizable we would be happy to push them to a less formal community plugin visualization library that users can pick/choose what types to install. I would greatly recommend such a "visualization library" be created to allow folks to contribute a wider variety of charts. No matter what library you go with, you will get a much larger variety of use cases covered if you harness the open source community to contribute new chart types.

mistercrunch commented 4 years ago

The thing with plugins is that you're free to do just that. For now the proper plugin framework is very recent, and the "plugin ecosystem" hasn't had the time to emerge just yet.

Everyone is free to start their own repo and publish plugins, it becomes fairly easy for anyone to build whomever else's plugin or sets of plugins into their Superset instances. The core team has decided directionally to standardize on ECharts as the main library to help us achieve more visual and technical consistency, and that applies to the the core plugins managed and maintained by the committers. Historically it's been challenging for Superset to be a patchworks of visualization frameworks and libraries that look slightly different, have different APIs, and evolve [or stagnate!] at different paces.

Longer term, we'd love to have a proper plugin ecosystem with one-click installs. We have a proof of concept somewhere that we can share, where you can point your installation to a built bundle (inside a CRUD view listing out your plugins) and it hot loads it and does the proper thing.

We may consider promoting plugins or plugin sets into the core plugins that ship with Superset too, but I'd much rather working towards a frictionless ecosystem that fosters distributed maintenance and ownership.

mistercrunch commented 4 years ago

Directionally, I vote against brining "Incorporating other map sources in Superset" if "incorporating" means bringing alternate map plugins to the main project, that the core team has to maintain/stabilize/evolve/release.

We could be doing a better job on the Deck-GL-based plugins, and the fact that we're not doing this as well as we should screams that we shouldn't bring in an alternate approach to geospatial.

BUT! we're super happy to work on the plugin ecosystem, by enabling things like:

having a page on the website that points to all existing plugins
make it easy to discover and install plugins inside Superset
improve the plugin interface to support more use cases and make it easier for people to author and publish plugins
....

roaur commented 4 years ago

That sounds fair. Between the Hello World blog post (thanks a lot, @villebro) and ECharts examples could a JS newbie implement the leaflet EChart example into Superset? Could you point me in other places you'd look if you were to implement such a chart into Superset?

juen1jp commented 4 years ago

Keeping the alternate plugins outside seems like the smart way to go for me too. Perhaps we should instead close this and open another SIP focused on formalizing the plugin ecosystem (although I am assuming that is already on the roadmap)

MihailGershkovich commented 3 years ago

we'll start to investigate those questions soon. Probably there will be some stuff that we can provide.

sfkeller commented 3 years ago

Keeping the alternate plugins outside seems like the smart way to go for me too. Perhaps we should instead close this

I'd propose not to close this and to get the focus back on this SIP, which is incorporating another map library instead of Mapbox GL JS (which in turn accesses a freely accessible base map server).

Specifically it's about replacing "deckl.gl Scatterplot".

I would not take WMS as service protocol but stick to the raster tiles (XYZ) scheme, like Mapbox base maps - now based on free tile servers, like http://openwhatevermap.xyz shows.

And to clarify this: You only need to exchange the Mapbox GL JS map library with Leaflet. You don't necessarily need to host the map tiles yourself.

mistercrunch commented:

We recently made the resolution to use ECharts as our primary visualization library #10418

Do the maps in ECharts depend on mapbox?

There are about four map types at https://echarts.apache.org/en/download-extension.html , among them Mapbox - and echarts-leaflet.

echarts-leaflet is the one we want: https://github.com/gnijuohz/echarts-leaflet

Instead of a map token this would need a urlTemplate like 'https://{s}.tile.openstreetmap.fr/hot/{z}/{x}/{y}.png'

So: What's left to be done in order agree on this and to implement this?

rusackas commented 3 years ago

Due to a numbering snafu, there was another SIP that was numbered SIP-53. While this issue is older, and really had the right to that number, the other one has already been voted through, so it should not be renumbered. I'll move this one to the next available SIP number, which is SIP-63. Sorry for the extra layer of awkwardness.

septatrix commented 2 years ago

An important feature for us would be synthetic maps (e.g. seat plans in restaurants) over which one can layer polygons in colors depending on the value

igorwiese commented 2 years ago

Hi @villebro, How is the progress on this SIP? Is it approved? Is anyone working on this? Thank you! :)

Doctor-Who commented 2 years ago

Maplibre integration or DeckGL enhancement as a true opensource project should be considered. Geographic preview and live maps are really useful on many dashboard made with superset.

rusackas commented 1 year ago

This sip has not been put up for a [VOTE] thread on the dev@ mailing list yet. Is there any intent to carry that forward? I'm not sure the SIP is actually needed, since this is effectively about building additional plugins. We can close this out if the author and others agree.

Stephane-Thales commented 1 year ago

Can this be kept alive until a solution is found ? Can the it be put to a vote ? Or is there a place to link-it for additional plugins request ?

rusackas commented 1 year ago

Well, I have a few thoughts on the matter, personally, but curious to hear the opinions of others (including @gk1089)

Is the intent here to add additional mapping plugins? If so, I don't think a SIP is required.
Is the intent here to replace Mapbox? This is more contentious, and would need a SIP.
If the idea is to add additional plugins, but the author doesn't have the intent to contribute them directly, this should probably be filed under an "Ideas" thread in the repo's Discussions.
If the idea is that someone's taking this project under their wing, we greatly look forward to a PR, and would be happy to help with any tactical questions along the way either on the PR itself or in the #contributing channel of the Slack workspace.

Wondering if anyone (dis)agrees with the above, and we can decide what to do with the SIP status accordingly.

Stephane-Thales commented 1 year ago

Thanks Evan, In my point of view : add additional mapping plugins But I've no (skill) capacity to do that myself.

villebro commented 1 year ago

I agree with @rusackas on the bullet list. The way I see it, I also agree it'd be great to add support for additional map sources. IMO we should have a MAP_SOURCES or similar config into which one could add entries for map sources that geospatial charts could use. Something like

class MapSource(str, Enum):
    MAPBOX = "mapbox"
    LEAFLET = "leaflet"
    OPEN_LAYERS = "open_layers"
    ....

MAP_SOURCES: Dict[MapSource, Dict[str, Any]] = {
    MapSource.MAPBOX: {
        <mapbox params go here...>
    },
    MapSource.LEAFLET: {
        <leaflet params go here...>
    }
}

This info would then be passed along to the frontend in bootstrap data, and viz plugins would then have a dropdown where the end user could choose which of the enabled map sources they want to use for a specific geospatial chart. And if the config is empty, geospatial viz' would be completely disabled.

gk1089 commented 1 year ago

I am happy that this is still being considered. The following has been my reasoning behind initiating this issue and I think it still stands:

The mapbox chart extension, while very easy to use, is not in the true spirit of open source. It needs registration and requires a key. It does not support overlaying other layers that a user might find more relevant to their use case.
Overlaying WMS services as layers of choice can be a powerful way of customized representation. This is supported by many popular BI tools and should be a part of default charts and not left for the users to develop from scratch.
There are pretty straightforward and widely used JS libraries that make embedding a map possible, like Leaflet and OpenLayers.
With my limited exposure to development for Superset, I think creating a chart for maps is more involved than other kinds of charts. Even the superset suggested charting library ECharts does not have support for WMS layers so there is even less building blocks to work upon.

I stand behind the suggestions of @rusackas and @villebro though to begin with even the option to add a WMS layer would be a great start since these can be easily published with customizations using widely available open source tools.

I have found Superset very useful and am willing to take up the challenge with some students that come work with me periodically. But the main challenges I face are a very restrictive internet environment that prohibits the use of many live discussions platforms. And even more than that, the lack of authoritative and updated information regarding starting from ground up and reaching the latest development build. If someone can give a focused push in getting us running, I am sure we can come up with something useful in the coming six months.

Cheers!

rusackas commented 1 year ago

@gk1089 I think if we're not replacing the mapbox plugin, but either adding new ones, or (better yet) making the current one more extensible as @villebro suggests, we can probably close this as an issue and cancel the SIP, and instead convert this to an "Ideas" discussion. Let me know if you have any objection.

Also, I'm happy (as is Ville, I'm sure) to find a way to collab with you and your students on this - are you able to join the Slack community or a Zoom call, when the time is right? If not, please feel free to email me and we'll sort something out.

Stephane-Thales commented 1 year ago

Hi @rusackas and @gk1089 just checking in. Any news regarding this feature ?

gk1089 commented 1 year ago

Hi @rusackas and @gk1089 just checking in. Any news regarding this feature ?

I am working with my students and getting them up to speed is taking time. I hope to have some positive outcome in the next 3 months. I will definitely reach out to @rusackas & @villebro when I get stuck eventually.

Doctor-Who commented 1 year ago

Many thanks for your work @gk1089 perhaps should consider maplibre instead using mapbox ? no key required, same powerfull features, maplibre is growing fast.

Stephane-Thales commented 1 year ago

I've seen that Apache has took over Echarts, that contain lots of geo visualisation. I can't find if they are using Mapbox or another paying provider it look like it's fully community based ? So it's also an option ?

villebro commented 1 year ago

@Stephane-Thales I suggest you check out this discussion: #21758

Stephane-Thales commented 1 year ago

@Stephane-Thales I suggest you check out this discussion: #21758

Thanks, interesting and there are perhaps some overlaps, but that's not exactly what we are discussing here.

w4rell commented 1 year ago

Hi there,

Any news @gk1089 ? It would be awesome to be able to put maplibre links or have the opportunity to change the map tiles provider ;)

Thanks,

Doctor-Who commented 1 year ago

It seems there is a plugin with openlayers but don't know how to install it :persevere: https://www.npmjs.com/package/superset-ol-plugin

Stephane-Thales commented 1 year ago

It seems there is a plugin with openlayers but don't know how to install it 😣 https://www.npmjs.com/package/superset-ol-plugin

Hi @Doctor-Who it's not part of the core superset so indeed it looks difficult to setup. You need to recompile superset if I understood correctly. Also, there is no descriptions of the features.

jansule commented 1 year ago

It seems there is a plugin with openlayers but don't know how to install it persevere https://www.npmjs.com/package/superset-ol-plugin

I am currently working on moving the openlayers plugin to the core. I just have to fix some build issues, but I am sure that I will be able to open a first PR soon.

w4rell commented 1 year ago

It seems there is a plugin with openlayers but don't know how to install it persevere https://www.npmjs.com/package/superset-ol-plugin

I am currently working on moving the openlayers plugin to the core. I just have to fix some build issues, but I am sure that I will be able to open a first PR soon.

That's a really good news! Thanks for your dedication!

Stephane-Thales commented 1 year ago

It seems there is a plugin with openlayers but don't know how to install it persevere https://www.npmjs.com/package/superset-ol-plugin

I am currently working on moving the openlayers plugin to the core. I just have to fix some build issues, but I am sure that I will be able to open a first PR soon.

Thanks ! keep us posted here.

Doctor-Who commented 1 year ago

Many thanks for you hard work !

kashifjavedaddo commented 1 year ago

@jansule, Can you please update on the build issues regarding superset-ol-plugin. Have they been fixed?

I have followed these steps so far:

cd /opt/superset/superset-frontend nvm use 16.19.1 npm i superset-ol-plugin

_npm ERR! code ERESOLVE npm ERR! ERESOLVE unable to resolve dependency tree npm ERR! npm ERR! While resolving: superset@0.0.0-dev npm ERR! Found: @ant-design/icons@5.2.6 npm ERR! node_modules/@ant-design/icons npm ERR! @ant-design/icons@"^5.0.1" from the root project npm ERR! peer @ant-design/icons@"^5.0.1" from @superset-ui/chart-controls@0.18.25 npm ERR! packages/superset-ui-chart-controls npm ERR! @superset-ui/chart-controls@0.18.25 npm ERR! node_modules/@superset-ui/chart-controls npm ERR! @superset-ui/chart-controls@"0.18.25" from @superset-ui/legacy-plugin-chart-time-table@0.18.25 npm ERR! packages/superset-ui-demo/node_modules/@superset-ui/legacy-plugin-chart-time-table npm ERR! peer @superset-ui/legacy-plugin-chart-time-table@"" from @superset-ui/demo@0.18.25 npm ERR! packages/superset-ui-demo npm ERR! 26 more (@superset-ui/legacy-plugin-chart-time-table, ...) npm ERR! 1 more (@superset-ui/plugin-chart-pivot-table) npm ERR! npm ERR! Could not resolve dependency: npm ERR! peer @ant-design/icons@"^4.2.2" from superset-ol-plugin@0.6.0 npm ERR! node_modules/superset-ol-plugin npm ERR! superset-ol-plugin@"" from the root project npm ERR! npm ERR! Fix the upstream dependency conflict, or retry npm ERR! this command with --force, or --legacy-peer-deps npm ERR! to accept an incorrect (and potentially broken) dependency resolution. npm ERR! npm ERR! See /home/asupset/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in: npm ERR! /home/asupset/.npm/_logs/2023-09-21T13_57_33_253Z-debug-0.log__

Please note that I also tried with npm i superset-ol-plugin -- force and npm i superset-ol-plugin --legacy-peer-deps and was able to proceed to next step i.e. npm cli and again had to do npm cli -- force but after that npi run build keeps giving errors.

Looking forward to your reply.

rusackas commented 1 year ago

This SIP has not been brought up for a vote, and the thread is a bit all over the map - I'm not quite sure the scope of the standing proposal that we'd be voting on. Does anyone feel like recapitulating the thread/proposal and calling a vote, or should we abandon this and start a new proposal when people have availability to pursue the renewed effort?

Stephane-Thales commented 1 year ago

This SIP has not been brought up for a vote, and the thread is a bit all over the map - I'm not quite sure the scope of the standing proposal that we'd be voting on. Does anyone feel like recapitulating the thread/proposal and calling a vote, or should we abandon this and start a new proposal when people have availability to pursue the renewed effort?

Hi @rusackas , for me the initial description of the issue is still relevant, i don't think the thread changed the scope. Can we reuse it ?

jansule commented 1 year ago

A first PR for supporting cartodiagrams and external map sources (WMS, WFS, XYZ) was created: https://github.com/apache/superset/pull/25869

apache / superset