Speed up map in category's description

ThomasDaheim commented 8 years ago

Hi,

with 7300+ pictures with coordinates and 100 gpx tracks the map in the category's description takes quite some time to load (see http://www.feuster.com/piwigo/).

Is there any way to speed up things here? In my previous gallery (see bilder.feuster.com) I had only a marker per category and didn't show the tracks. Is something similar possible here?

Thanks, Thomas

ThomasDaheim commented 8 years ago

Might be a bit hard to implement but this looks like a good idea: https://www.phase2technology.com/blog/large-scale-server-side-mapping-in-drupal-with-the-leaflet-geocluster-stack-part-1/

xbgmsharp commented 8 years ago

Hey, I check and yes it is slow. The questions is where is the slowness. In display the data or in fetching the data, in the SQL part of or the PHP. If you could add print some timestamp and therfor deduced where the slowness is happening. All data are per-calculted ofr smooth navigation.

There is a few options:

If you don't use any of the optional fields, you can remove them from the SQL query.
Use SQL Views, therefor the data are already calculated. I started but never finish.

No sure which is best. Would require test.

ThomasDaheim commented 8 years ago

Hi, I have added a few outputs of microtime() in functions_map.php:

osm_get_items: start - 0.53629400 1448124445
osm_get_items: before query - 0.53640700 1448124445
osm_get_items: after query - 0.72951600 1448124445
osm_get_items: end - 0.82367200 1448124445
osm_get_gps: start - 0.83407900 1448124445
osm_get_gps: before array_from_query - 0.83417600 1448124445
osm_get_gps: end - 0.88539900 1448124445
osm_get_js: start - 0.88548100 1448124445
osm_get_js: end - 0.97030700 1448124445

After last timestamp there is still a lot of time spent before the map starts to paint. Where else would I need to add similar output?

ThomasDaheim commented 8 years ago

Well, after some playing around I would think the slow part is the rendering in the browser and not the php server part. Not sure how to put printing of timestamps in there?

xbgmsharp commented 8 years ago

Agreed, using the web developer tools with network, you can check the size and time of the page Maybe it is large and take time to transfer and to render in the broswer. You might want to compress HTML content.

ThomasDaheim commented 8 years ago

Now I have collected some info - but not sure what to do with it:

Web page as it is sent to the browser: 2.5MB with most of it made up by the definition of "addressPoints" and 69 lines of loading gpx via "omnivore.gpx"
har output from Firebug
json output from WebDeveloper Performance

xbgmsharp commented 8 years ago

How long did it take to transfer the webpage of 2.5M? I believe the slowless is the fact of loading the javascript library Leafelet and omnivore then inject the addressPoints.

ThomasDaheim commented 8 years ago

I have uploaded the output under http://www.feuster.com/piwigo/_debug/ - its probably easier if you have a look at it directly...

Loading the page takes about 1.6-1.8secs. After loading various js files there is a huge gap (2secs) before the tiles of the openstreetmap and the gpx files get loaded. Afterwards there is another gap before some more js, png, ... get loaded.

ThomasDaheim commented 8 years ago

Well, I have now helped myself with the following:

If the page doesn't show a category with pictures but only sub-categories => don't load gpx and load only first picture of each category

Speeds up things a lot :-)

When I find more spare time I might try to add this as a new setting instead of hard-coding it into functions_map.php...

patmans commented 8 years ago

I hit this slowdown too, with about 288 GPX files, and (so far) only 3 photos with GPS information.

Both firefox and chrome use lots of CPU time and a lot of memory, and neither show anything for the map views even after waiting for about 20 seconds.

There's pretty much no load on the server side.

I disabled both the worldmap view and the "map in category's description", and then the pages all load OK, but I really like seeing the map views!

ThomasDaheim commented 8 years ago

I'm planning to complete my change and submit a pull request in the upcoming quiet time :-)

xbgmsharp commented 8 years ago

I beleive the slowness is coming from the GPX as they include a lot of multiple GPS points. I personally test on my dev box with 10.000 entries and no slowness but not GPX file. The list of data in the request should already be limited to the current category, excluding all other. As you mention loading load only first picture of each category and not the GPX that could help. I was more thinking to only load the first point of the GPX instead of all as GPX file can include many points. You mainly need to the starting point. on the categories page. Another option would be change leaflet plugin to leaflet-omnivore if any faster. It would allow to add kml-support form #70 and #72.

xbgmsharp commented 8 years ago

In fact it already use leaflet-omnivore. So we could create a fast option that would only parse the first point of each GPX file.

ThomasDaheim commented 8 years ago

Not sure if things would get faster a lot if you have to open the gpx to find the first point... Also, with support for kml you would need to learn to parse two tips of files.

How about that: since the gpx/kml is also an item in a category you could its position and show with some special marker/dot on the map?

In my case that doesn't seem to be enough since without gpx and 8300+ images it still takes long time to render the map. The items you currently show are those from the category and all sub-categories, right?

joubu commented 8 years ago

The function omnivore.gpx creates 1 ajax request. So if you have 250 gpx, you will generate 250 requests! A workaround would be to read the contain of the gpx files and use omnivore.gpx.parse instead.

xbgmsharp commented 8 years ago

After analyzing the HAR file and some test. The slowness come from the ajax requests. Those requests are not asynchronous therefor are blocking the rest of the page to be load. Each GPX file create an omnivore call:

omnivore.gpx('./upload/....22fb4762.gpx').addTo(map);

From https://github.com/mapbox/leaflet-omnivore#async--events

Functions that load from URLs are asynchronous...

patmans commented 8 years ago

For my GPX files (288 of them), the browser was not only slow but used well over 2 GiB of memory.

xbgmsharp - does your analysis account for that memory usage?

(And thanks ...)

joubu commented 8 years ago

I am currently working on something to improve this area. The idea is to generate a geojson file from the gpx, only once, instead of converting it for each query. You can also reduce the number of elements in your gpx file, using gpsbabel:

gpsbabel -i gpx -f file.gpx -x simplify,crosstrack,error=0.001k -o gpx -F file_filtered.gpx

ThomasDaheim commented 8 years ago

I have now added all my changes to speed up map rendering in my branch - now part of the outstanding pull request "Add kml-support"

Added option "hide gpx for sub-categories": if yes, only gpx for a category with own images is shown
Added option "show only first image for sub-categories": if yes, only the first image per sub-category is shown

Those two settings to yes reduces my map load time by an order of magnitude... Happy to test any other speed improvements as well!

Additional changes included in the commit:

Added option to select which target should be used when clicking on the image in a popup
Previous fix for #74

Have fun!

Piwigo / piwigo-openstreetmap

Speed up map in category's description #73