[request for help] getting lat/long bounds of visible map

[squirreliferous]

Hello,

I'm trying to develop a python web app (using Heroku and Flask) to displays data from a public API. The idea is to display the catalogued information of an organisation's assets by plotting their locations on a map. The information is displayed as a pop-up when the user clicks on the icon.

I have done this successfully using a local copy of the data set so I know that the scheme is practical from an aesthetic/usability stand point with FastMarkerCluster. However, the ~400k records are updated semi-regularly so using a local copy is not optimal and I'd rather pull data as required from the organisation's public API. I've tried to pull all the data from the API at once, but this maxes out the memory resources that Heroku assigns to my free account and the app crashes before it even loads.

To avoid this, I'm thinking to pull data from the API only after a zoom level threshold and only for the visible map area. The API appears to allow such queries, so I need to find a way to get the visible map bounds and update these on panning. Leaflet appears to have the functionality I need in getBounds(). From the docs, it appears that get_bounds() is the Folium version, but this example notebook shows that the function only returns the bounds of an object and not the whole map. Am I missing something here, or does the functionality I'd like not yet exist in Folium?

Thank you very much in advance and for all the work already done on Folium, Sorry if this isn't the right place to ask, but I already tried StackOverflow but haven't received a clear answer. -Rameses

[fullonic]

@ramesesjd I'm not sure, but I think what you are looking for isn't available on folium. At least, after reading more folium code I can't see anything that can do that for you. What do I understated from reading get_bound() and also _get_self_bounds() is that it won't create any javascript for you, is only python code.

From what you explain, probably the best way is using java script to get map view bounds and send the coordinates to the rest api. From the api response, update again your map. Can you share your public api, so I can test my idea before post any code?

[Conengmo]

You're right @fullonic, folium isn't really suitable for dynamic callbacks. folium is a one way street, generating HTML + Javascript from Python. There is no way in folium to get data back from Javascript to Python. But of course you can make such a thing yourself!

[Conengmo]

I'm closing the issue because it's not really actionable for us. But feel free to resume the discussion here. If you have ideas on how folium can be improved we can reopen the issue, or you can open a new one or a PR.

[squirreliferous]

@Conengmo, thank you; I'm fine with the issue being closed at this time. If you don't mind, I'd like to continue discussion with @fullonic in this thread and potentially re-open/submit a PR, depending on progress.

[squirreliferous]

@fullonic,

From what you explain, probably the best way is using java script to get map view bounds and send the coordinates to the rest api. From the api response, update again your map.

This is what I'm thinking as well, but I've got no java script experience at this point but am willing to learn. The public api is here. From the socrata docs, I was thinking to filter the data using an SoQL query with the map bounds then plot that data on the map. Not sure if this is feasible, or if it would be way too many calls to the API. I've signed up with the data provider and have an apptoken to allow me to access all the data; without the token the data is limited to ~1000-2000 rows.

Any assistance you can provide to get me on track would be very welcome. FWIW, here is the Folium map made using the local copy of the data set. Rather than serving up a huge, static webpage like this, it would be nice to have the data dynamically loaded, so it's up-to-date. Dynamically loading the data all at once overloads the Heroku server, at the free tier. It's possible the paid tiers would be fine with the load, but I'm just learning at present, so would rather do things cheaply.

[fullonic]

@ramesesjd I saw your maps, look really good but as you say, is a lot of data to render.

Do you know how many request can you do to the api? Because it would be a lot of requests using the approach of each-time-map-bounds-change/request.

Why not use the data from your local copy? It works, correct? And to solve the problem about outdated data, you can create a script that update's your local copy and run it periodically using a cron job. In Heroku is called Heroku Scheduler.

However during the week I will try to get time and present you a possible solution or workaround for the approach of dynamically update the map.

[squirreliferous]

@fullonic Not sure how many api requests can be made, and it's not clear anywhere on the Socrata website. You're right, it would be quite a lot of requests using the dynamic request approach.

Why not use the data from your local copy?

Do you mean make API calls to the local copy? That's actually quite a good idea which I hadn't thought about. I'll look into what's required to convert the local copy to a basic SQL database. It would make the page much smaller in size which would be good. Also, I didn't know about Heroku Scheduler, so thanks for that tip.

However during the week I will try to get time and present you a possible solution or workaround for the approach of dynamically update the map.

Thank you for this. Even going the route of API calls to the local database copy will need this, so I'm very grateful for any help you can provide.

[fullonic]

@ramesesjd making api calls to your local-copy/db was my first thought, however I'm not sure if with your free heroku tier will be enougth to get good results. Also, for that you will need something like PostgreSQL, and you can have one for free with heroku, but is limited to 10000 rows I think so.

Probably, what you can do for now, is keeping using the data from your local copy like you are doing now, and keep running a script to update your data as a cron job.

I will let you know when I'm done with a example. Stay well

[fullonic]

@ramesesjd I created a repo with an example of what you had asked about. It have alot of room to improve but it does the job. The data is rendered in base of the map bounds. However when I tried to run the server with all your data points, it doesn't works. (I took it from your repo to test it ;) ) See it by yourself: https://github.com/fullonic/flask_folium Isn't finished yet, but I don't have too much free time for now. I will keep adding stuff to the repo to improved it. Let me know what do you think.

[squirreliferous]

@fullonic Thank you very much for the quick response! I'll have a look at the repo. The coming days are quite full for me, so will not have much time till next week, but I will keep in touch with you, either here or on the repo directly. Thanks again!

[seghier]

I think this is possible because the minimap show exactly the bounds of the actual map