search

streetcomplete / StreetComplete

Easy to use OpenStreetMap editor for Android
https://streetcomplete.app
GNU General Public License v3.0
3.85k stars 351 forks source link

Performance improvements: Faster Overpass queries #1514

Closed westnordost closed 5 years ago

westnordost commented 5 years ago

Some Overpass queries are very slow and clog up the download process by pitching the app in a long queue, waiting to replenish its Overpass download quota. If the queries were executed faster, the app would reach the quota later.

So:

  1. Find out which quests take long and why. First step here is to create a little program in the tests/ directory that iterates through the quest type list and outputs the compiled overpass query for each for some bbox to console. See https://github.com/westnordost/StreetComplete/issues/1457#issuecomment-509031473. The output of this program would also be interesting for @rugk who is maintaining the quests wiki page and links to the used overpass queries for each quest type. Executing each query on some location should give an idea which quest types are problematic.

  2. Implement optimizations based on these observations. In the predecessor ticket to this one, a few suggestions have already been made:

    1. make possible the usage of regex in keys in the TQL syntax and use it where applicable
    2. express nested ors in OQL with if: <condition> blocks if they don't contain regex
    3. Far the biggest impact though has some fiddling of evaluation strategies for Overpass Queries. At the moment, it is unknown what the best strategy would be here. See https://github.com/westnordost/StreetComplete/issues/1457#issuecomment-513545692 and /cc @mmd-osm
westnordost commented 5 years ago

How can the dev instance be 12 times faster than lz4?

mmd-osm commented 5 years ago

It runs a different fork/branch with performance fixes.

westnordost commented 5 years ago

That's pretty incredible

On September 17, 2019 7:22:04 AM GMT+02:00, mmd notifications@github.com wrote:

It runs a different codeline with performance fixes.

rugk commented 5 years ago

It runs a different fork/branch with performance fixes.

So would not it be useful to contribute these back upstream? 😄

mmd-osm commented 5 years ago

Sure, that’s ongoing...

goldfndr commented 4 years ago

I've noticed that in the resulting regular expression syntax, some have the ORs in parentheses, most don't. For example, AddRoadName has both "^(primary|secondary|tertiary|unclassified|residential|living_street|pedestrian)$" and '^private|no$', and AddCycleway has them explicitly but AddWheelchairAccessBusiness doesn't have them. Is there a benefit to including the parentheses or detriment to avoiding them?

westnordost commented 4 years ago

Hm, I think ^private|no$ may be a problem. Did you find it only in AddRoadName, or in other places as well?

See https://regexper.com/#%5Eprivate%7Cno%24

westnordost commented 4 years ago

Right, the overpass query was generated without the parentheses. Should be fixed now.

westnordost commented 4 years ago

Anyhow, a big thanks for noticing and notifying me about it! That's quite a big bug!

rugk commented 4 years ago

Testing the AddRoadName query I am surprised that Overpass Turbo actually highlights me the streets with names and not without ones…

mmd-osm commented 4 years ago

NB: It seems this is entirely unrelated to the Performance optimization in this issue. The original queries already had the same issue. As an example: https://github.com/westnordost/StreetComplete/commit/6d234a438021288d7334f2b004dbe3016063cf46

goldfndr commented 4 years ago

Thanks. Any possibility if you could yet again update the final syntax?