Support for parallel tile downloads

[rinigus]

I am developing a simple server that can be used as a drop-in replacement for online map services: https://github.com/rinigus/osmscout-server . The integration with poor-maps was very simple (see poor-maps under thirdparty subfolder). However, it would be great to be able to specify how many downloads can be performed simultaneously. It seems, that poor-maps is downloading using only single connection. While the speed is reasonable in the interaction with the local server, I think it would be faster if I could generate 4 tiles in a time (as many as CPU cores on the phone).

I'd suggest to add "threads" or some similar option to tilesourses/ .json files, for example. It could 1 as a default and changed if needed.

[otsaloma]

I like the sound of what you're working on!

It seems, that poor-maps is downloading using only single connection. While the speed is reasonable in the interaction with the local server, I think it would be faster if I could generate 4 tiles in a time (as many as CPU cores on the phone).

Poor Maps currently uses two download threads. That number has been chosen to match what is allowed by the OpenStreetMap tile usage policy. I don't want to add a field to the JSON files as that could encourage misuse and get our user agent banned. But, what I could do, is make an exception in the tile source code for localhost sources. Is it OK for the thread count to match the logical processor count or should it be one less? Does your server run at normal priority? Will using all cores make the UI lag?

And regarding integration, it would be fine by me if your RPM installs JSON files to /usr/share/harbour-poor-maps/tilesources. Poor Maps uses whatever happens to be there. It would make it zero-config for users. But do name the files to match existing conventions -- including the "x" i.e. osmscoutday@1x.json.

[rinigus]

Thank you for encouragement :) . I do understand why you don't want to make such field. At present, I don't know whether using many threads would make UI lag. Its not too slow right now either... Maybe I should experiment a bit on that on phone by hacking poor-maps and changing the number of threads. Where I can do that?

There will be problems with getting logical number of CPUs on phones. Namely, kernel (or user-land daemon) are actively switching them off. So, calls to Qt function that is supposed to tell you the number of optimal threads do return almost a random number from 1-ncores.

As for integration, your solution is simple (and sorry for missing x, I am sure I had that before!). I'll think about it - it may lead to multiple problems in future though (its hard to remove files in rpm upgrades).

I'll work on implementing search layer and, hopefully, routing and would come back here when I have done some testing on threads.

PS: its so simple to add new sources in poor-maps, thank you for making it so easily extendable! and a joy to use your program!

[otsaloma]

Maybe I should experiment a bit on that on phone by hacking poor-maps and changing the number of threads. Where I can do that?

TileSource._pool: https://github.com/otsaloma/poor-maps/blob/master/poor/tilesource.py#L43

Easiest for testing is probably just to SSH to the device and edit the installed file in-place.

$ devel-su
# nano /usr/share/harbour-poor-maps/poor/tilesource.py

There will be problems with getting logical number of CPUs on phones.

So it seems. But that can probably be worked around -- Android folks seem to look under /sys/devices/system/cpu/.

I'll work on implementing search layer and, hopefully, routing and would come back here when I have done some testing on threads.

Sounds good. There's been questions about offline since day one when releasing Poor Maps 0.1. It's good to see a modular solution for the problem.

[rinigus]

Thank you! I'll play with it a bit later when I get other search and routing working.

Re CPUs: That's true - I am doing the same in https://github.com/collectd/collectd/pull/1823 (corresponding code around https://github.com/rinigus/collectd/blob/sailfish/src/cpufreq.c#L203 ). The folder is still there even if CPU is switched off. I guess I would have to add this code into the server and we could test later something similar in poor-maps.

[otsaloma]

Let me know if the download thread count works now as expected. I wrote my own HTTP connection pooling code for Poor Maps to avoid a dependency not available via Sailfish repos (e.g. requests) -- I'm not a 100% sure it works right.

[rinigus]

Thank you very much! I think it works as intended. The speed is considerably faster :)