Add a script version of MarkerCluster to speedup creating many markers.

[ruoyu0088]

For example following code costs 5 seconds to add 1000 markers:

import folium
import numpy as np
from folium.plugins import MarkerCluster
N = 1000
fig = folium.element.Figure()
map_ = folium.Map(location=[20, 50], zoom_start=3)
cluster = MarkerCluster(np.random.uniform(0, 90, (N, 2))).add_to(map_)
map_.add_to(fig)

and the output html contains code for 1000 markers.

I tried a Javascript version to speedup this:

class MarkerClusterScript(MarkerCluster):
    def __init__(self, data, callback):
        from jinja2 import Template
        super(MarkerClusterScript, self).__init__([])
        self._name = 'Script'
        self._data = data
        if callable(callback):
            from flexx.pyscript import py2js
            self._callback =  py2js(callback, new_name="callback")
        else:
            self._callback = "var callback = {};".format(_callback)

        self._template = Template(u"""
{% macro script(this, kwargs) %}
(function(){
    var data = {{this._data}};
    var map = {{this._parent.get_name()}};
    var cluster = L.markerClusterGroup();
    {{this._callback}}

    for (var i = 0; i < data.length; i++) {
        var row = data[i];
        var marker = callback(row);
        marker.addTo(cluster);
    }

    cluster.addTo(map);
})();
{% endmacro %}
            """)

and then create the map by

import pandas as pd
N = 1000
df = pd.DataFrame(np.random.uniform(0, 90, (N, 2)), columns=list("xy"))
df["color"] = np.random.choice(["red", "green", "blue"], N)

fig = folium.element.Figure()
map_ = folium.Map(location=[20, 50], zoom_start=3)

def create_marker(row):
    icon = L.AwesomeMarkers.icon({markerColor: row.color})    
    marker = L.marker(L.LatLng(row.x, row.y))
    marker.setIcon(icon)
    return marker

MarkerClusterScript(df.to_json(orient="records"), callback=create_marker).add_to(map_)
map_.add_to(fig)

It only takes 0.3 seconds. All the data are saved in a javascript array, and the for-loop for creating markers occured in web brower.

I suggest to add this kind of Classes for speedup.

[ocefpaf]

@ruoyu0088 thanks for the suggestion! Feel free to send a PR addressing that. Right now we are focusing on fixing bugs and cleaning the code for a new release. Optimizations are not high priority.

[eddies]

@ruoyu0088 that would be brilliant. I only just realized last week when trying to plot > 5000 markers how slow it could be.

[BibMartin]

generalities

I'm okay with @ocefpaf on:

Optimizations are not high priority.

In fact, folum's goal is to provide a simple way of creating maps. For that, we need to introduce a kind of modularity that often prevent optimisations.

People asking for optimisation have to take the effort of going into javascript, or write their custom class (as you did).

in that particular case

In the particular case of MarkerCluster we've chosen the current implementation because it enables you to tune the markers when adding them one after the other. I can (for example) do a MarkerCluster combining different markers, with different shapes/colors/popups, etc.

But I agree this does not address everyone's need and it would worth having a second MarkerCluster object for people who wants to go faster. So please make a PR.

Instead of MarkerClusterScript, I would suggest to name it FastMarkerCluster.

btw : I really like the way you've implemented it (with the callback).

[JamesGardiner]

@ruoyu0088 your class was incredibly useful for plotting 25,000+ points globally http://nbviewer.jupyter.org/gist/anonymous/33f38b09ad3bfa277e2d9c06e4bb588c

[ocefpaf]

Cool use case @JamesGardiner. If @ruoyu0088 does not mind maybe you could send the FastMarkerCluster PR suggested in https://github.com/python-visualization/folium/issues/416#issuecomment-209361066. (With the proper credits to @ruoyu0088 of course!)

[HFulcher]

Relating to this issue I have approx 147k points I want to add to a map. I realise this is a very large amount but it would be useful if I could have an update on whether there is an efficient way of placing these markers on the map as clusters. Currently the standard way takes a long time (understandably). Thanks

[ryanbaumann]

@JamesGardiner the geojson-vt utility in Mapbox gl-js is pretty powerful. I'm starting work to add Folium support for GL maps - in the meantime have a look at the Mapbox GL JS SDK!

[JamesGardiner]

Have had a chance to look at this recently - I've got the FasterMarkerCluster script suggested by @ruoyu0088 working but need to make a few more changes. I'd like to avoid depending on flexx.pyscript.py2js (which itself depends on Tornado) and construct the callback using Jinja alone. Should be easily done though. Hope to issue a pull request this weekend.

[leblancfg]

Sorry for referencing a closed issue, but this is way above my Javascript capacities. Would there be an easy way at all for markers created using FastMarkerClusters to have popups as well?

[bukowa]

@leblancfg did you find the solution?

[leblancfg]

I ended up following what @JamesGardiner had done in the notebook he linked to previously:

# nan values throw the JS script
# Popups with name strings
popups = df.name[df.lng.notnull()].values

# Latitude and longitude dataframe with no nan values
locations = [list(a) for a in zip(lat, lng, popups)]
df_locations = pd.DataFrame(locations, columns=['lat', 'lng', 'names'])

fig = folium.element.Figure()
map_orgs = folium.Map(location=[56, -3], zoom_start=5)
MarkerClusterScript(df_locations.to_json(orient="records"), callback=create_marker).add_to(map_orgs)
map_orgs.add_to(fig)

The main dataset I work with has ~8k locations, and it works all right.

[pieterjoanUvA]

I created a hack to the FastMarkerCluster, but now I can create custom popups(labels) for all my markers with different map-icons, (also tried the code above, and that does not function in folium 0.7 )

The ugly part is first pass a list with 3 arguments, then in the class satisfy the super-conditions and create that second list, that is appointed to self._popup .

# popups 
class FasterMarkerCluster(MarkerCluster):  
    _template = Template(u"""
            {% macro script(this, kwargs) %}
            var {{ this.get_name() }} = (function(){
                {{this._callback}}
                var data = {{ this._data }};
                var popup = {{ this._popup }};
                var cluster = L.markerClusterGroup({{ this.options }});
                var category1 = L.layerGroup();
                var category2 = L.layerGroup();

                for (var i = 0; i < popup.length; i++) {
                    var row = data[i];
                    var marker = callback(row);
                    var row2 = popup[i];

                        if (row2.indexOf("during") !== -1) {
                            var icon = L.AwesomeMarkers.icon({icon: "map-marker", markerColor: "red"});
                            marker.setIcon(icon);
                            marker.bindPopup("<h4>"+row2+"<h4>");
                            marker.addTo(category1);
                        } else {
                            var icon = L.AwesomeMarkers.icon({icon: "map-marker", markerColor: "blue"});
                            marker.setIcon(icon);
                            marker.bindPopup("<h4>"+row2+"<h4>");
                            marker.addTo(category2);
                        }                   
                }
                cluster.addLayers(category1);
                cluster.addLayers(category2);
                cluster.addTo({{ this._parent.get_name() }});
                return cluster;
            })();
            {% endmacro %}""")

    def __init__(self, data, callback=None, options=None,
                 name=None, overlay=True, control=True, show=True):

        super(FasterMarkerCluster, self).__init__(name=name, overlay=overlay,
                                                control=control, show=show,
                                                options=options)
        namesrow = data[:]
        data = [row[0:2] for row in data]
        self._name = 'FasterMarkerCluster'
        self._data = _validate_coordinates(data)
        self._popup = namesrow

        if callback is None:
            self._callback = """
                var callback = function (row) {
                    var icon = L.AwesomeMarkers.icon();
                    var marker = L.marker(new L.LatLng(row[0], row[1]));
                    marker.setIcon(icon);
                    return marker;
                };"""
        else:
            self._callback = 'var callback = {};'.format(callback)

with the finlist and call to this class:

sanfran_map = folium.Map(location = [latitude, longitude], zoom_start = 12)

## callback_template for customised icons --> also available in the normal FastMarkerClusters 
callback = """\
function (row) {
    var marker;
    marker = L.marker(new L.LatLng(row[0], row[1]));
    return marker;
};
"""
# instantiate a mark cluster object for the incidents in the dataframe
unis = plugins.MarkerCluster()

# loop through the dataframe and add each data point to the mark cluster
finlist = []
for mainCat, venueName, venueShort, uni_loc, lat, lng in zip(uni_venues['main_category']\
                                 , uni_venues['Venue'], uni_venues['Venue Category Short']\
                                 , uni_venues['uni_loc']\
                                 , uni_venues['Venue Latitude']
                                 , uni_venues['Venue Longitude']):
    labelfinal ='<h4>'+ mainCat+'</h4><h6>Name: '+venueName+'</h6>Cat.ShortName  : <i>' \
                + venueShort+'</i><h6> nearby Location(s) : '+uni_loc+'</h6>'
    finlist.append(labelfinal)

# Calls the customised class function.  
location_list_uni = list(zip(uni_venues['Venue Latitude'], uni_venues['Venue Longitude'], finlist))

# Append the finlist with the crime entries,
finlist2 = []
for label,dayname,timeofday in zip(df1['Incident Subcategory']\
                               , df1['Incident Datetime'].dt.day_name(), df1['timeofday']):
    labelfinal ='<h4>'+ label+'</h4><h6>'+'</h6>on a  : <i>'+dayname+'</i> during <b>'+timeofday+'</b>'
    finlist2.append(labelfinal)

location_list_crimes = list(zip(df1.Latitude, df1.Longitude, finlist2))

# loc_large = location_list_crimes[:] + location_list_uni[:]
# add the returned markerCluster-objects to the map.
crimes_cluster = FasterMarkerCluster(data=location_list_crimes,callback=callback)
venues_cluster = FasterMarkerCluster(data=location_list_uni,callback=callback)

# add the locations to the map in red-blue-translucent circles.
for lat, lng in zip(df_uni['latitude'],df_uni['longitude']):
    unis.add_child(
        #folium.features.CircleMarker( #### only 0.5 version of folium
        folium.CircleMarker(
            [lat, lng],
            radius=10, # define how big you want the circle markers to be
            color='red',
            fill=True,
            fill_color='blue',
            fill_opacity=0.6
        )
    )

# Create FeatureGroups to name the Layers, add the MarkerCluster-objects to these layers.
uni_group = folium.map.FeatureGroup(name='Universities', overlay=True, control=True, show=True)
crime_group = folium.map.FeatureGroup(name='Crimes', overlay=True, control=True, show=True)
venues_group = folium.map.FeatureGroup(name='Venues', overlay=True, control=True, show=True)

for lat, lng, label in zip(df_uni['latitude'],df_uni['longitude'],df_uni['name']):
    folium.Marker([
        lat, lng], 
        popup='<h4>'+label+'</h4>',
        icon=folium.Icon(prefix='fa',icon='university',color='green')
    ).add_to(uni_group) #.add_to(sanfran_map)

# Add the MarkerCluster and FasterMarkerCluster Objects to the FeatureGroups  
unis.add_to(uni_group)
crimes_cluster.add_to(crime_group)
venues_cluster.add_to(venues_group)
# Finally add the FeatureGroups to the map.
uni_group.add_to(sanfran_map)
crime_group.add_to(sanfran_map)
venues_group.add_to(sanfran_map)
folium.LayerControl().add_to(sanfran_map)

# display map
sanfran_map

just my try to fix this. and it works for my purposes .... :-)

[SamPse]

is it possible to use a custom icon as a parameter in the function def create_marker(row): icon = L.AwesomeMarkers.icon({markerColor: row.color}) marker = L.marker(L.LatLng(row.x, row.y)) marker.setIcon(icon) return marker MarkerClusterScript(df.to_json(orient="records"), callback=create_marker).add_to(map_) map_.add_to(fig) thank you