[wish] output compression

[edes]

Hello! Use case: Cycling & tracking around the world :) ...That makes quite some data, even with (Douglas-Peucker-)optimized tracks, and loading times get rather long - especially in regions without optimal Internet access. At first glance the output from gettrack seems to be uncompressed. Any way for output compression of that data?

[tinuzz]

Hi!

While I agree that compression sounds reasonable for 'gettrack' data, I'm not sure what I can do for you in Trackserver's code. I know it is possible to have PHP compress the output and send the appriopriate headers, but I think it is much easier to implement in the webserver. For example, in Nginx you can just set:

gzip on; gzip_types application/json;

and the output will be compressed if the client sends "Accept-Encoding: gzip" in the request.

If you see it differently, what would you like to see from Trackserver?

Cheers, Martijn.

[edes]

Hi Martijn. Just had a closer look into the json that contains the tracks information - it seems to be comressed already although the webserver is not configured to compress that kind of content type: Our GPX tracks weigh roughly 10 MB (in 80+ track files) on our local harddisk, but the json file is less than 400 kB - so I guess I was following the wrong trail here.

Nevertheless it takes the server 7-8 seconds to deliver the tracks to the browser (see http://away.edes.at/live-track). So, what I misunderstood as a data volume issue (it felt worse with lesser bandwith) must in fact be internal processing time issue.

The problem we have is, that we can't further reduce the number of data points in the GPX files without reducing the track's accuracy in an barely acceptable way. Before we go down the (rather time-consuming) road to joining the GPX tracks now - is there anything else we can do to reduce the server side processing time?

Thanks in advance! Hannes

[tinuzz]

Hi Hannes,

The "compression" that you see in the JSON is actually the polyline encoding (https://developers.google.com/maps/documentation/utilities/polylinealgorithm), which is the most efficient way of delivering track data (without timestamps!) that I know. Not much to gain there.

I agree that 7 seconds to deliver that little data (it's only 311 kB) is a bit long ;-) I see that it's 95 tracks. I haven't checked the number of track points, but I imagine is't quite a few. Constructing the data to send to the browser is done in 2 steps: a single SQL query, and the polyline encoding, which is a loop that constructs an array of all the points before feeding it to the encoder. I suspect the problem is in step 2, but I am curious how long the SQL query takes. Can you run the following query directly on your database? It should be the exact query that Trackserver runs when serving your map. (Change the table names if you use a non-standard prefix).

SELECT trip_id, latitude, longitude, altitude, speed, occurred, t.user_id, t.name, t.distance, t.comment FROM wp_ts_locations l INNER JOIN wp_ts_tracks t ON l.trip_id = t.id WHERE trip_id IN (7,35,38,36,6,7,8,16,17,11,18,19,21,22,23,34,40,42,44,43,45,46,48,50,55,57,58,60,61,63,62,65,64,66,67,68,71,70,74,140,77,76,79,78,81,83,85,87,88,89,90,91,93,95,94,97,100,99,98,101,102,103,106,105,104,107,109,108,113,112,114,115,117,116,119,120,121,123,125,126,128,129,130,132,139,138,137,135,142,146,143,148,54,149) AND l.hidden = 0 ORDER BY trip_id, occurred

On my server, this query returns around 8000 locations (probably less than on yours), but it only takes 0.15 seconds to run.

In the mean time, I'll think about it and maybe try to optimize the encoding step a little...

Cheers, Martijn.

[edes]

Hi Martijn The query returns 74500 datasets and took 0.2620 seconds to execute. Cheers Hannes

[tinuzz]

That's what I thought. I will have to optimize the code, but I'm not sure how, yet..

[tinuzz]

Hi,

Trackserver 4.2.2 was released today, and maybe it can help you with this issue.

After updating to the latest version, please visit the Options page and scroll all the way to the bottom. There you will find a checkbox for a setting called 'Single request mode', which is enabled by default.

If you disable it, Trackserver will switch back to an older way of getting track data and use 1 HTTP request per track, instead of 1 HTTP request for all tracks together. In your case, it might speed things up because your browser will get the tracks in parallel.

Can you try and let me know what effect it has for your website?

Best regards, Martijn.

[edes]

Hey Martijn First of all thanks a lot for coming back to this topic. I installed the latest version and tried it using Firefox on Android and with Chrome and Firefox on a Linux Desktop (all up to date versions) with different results: While loading the site (http://away.edes.at/live-track) in Firefox/Android I can see multiple pop-ups on the initial map saying "Track could not be loaded: 500 Internal server error. Re-activating the single request mode solves the problem. On the Linux Desktop (both Chrome and Firefox) the site is behaving like before disabling the single request option: The map is shown at an initial (close) zoom level, and the browser is indicating that the site is still loading. After all Tracks are loaded the map zooms to it's (very distant) final zoom level and shows the whole track in an instant. I'll keep the single request mode deactivated for a while so you can verify the results and see what I mean. Best regards Hannes

[tinuzz]

In late 2019, in commit 6611567, I changed the way that the polyline encoder is used in Trackserver. Before, it was an unmodified external library, that needed an array of coordinates as input. This required two steps to produce the track: once to turn the database result into an array, and a second step to turn that array into a polyline. I internalized the polyline encoder into Trackserver, so I could skip the step of producing the array, and generate the polyline directly from the database result. That should give about 50% performance increase.

Since this issue is already quite old, and a lot has changed since then, I'm closing it. If you still have concrete performance issues with Trackserver, feel free to re-open, or open a new issue.

Cheers, Martijn.