opq cannot take QL queries

Robinlovelace commented 7 years ago

To my knowledge (very happy to be proven wrong).

QL could be an ideal language for representing complex queries in text I think: see QL Basics. Illustrative examples (I don't think we can currently do this - any thoughts):

  ["key"]            /* filter objects tagged with this key and any value */
  [!"key"]           /* filter objects not tagged with this key and any value */
  ["key"="value"]    /* filter objects tagged with this key and this value */
  ["key"!="value"]   /* filter objects tagged with this key but not this value */
  ["key"~"value"]    /* filter objects tagged with this key and a value matching a regular expression */
  ["key"!~"value"    /* filter objects tagged with this key but a value not matching a regular expression */
  [~"key"~"value"]   /* filter objects tagged with a key and a value matching regular expressions */
  [~"key"~"value",i] /* filter objects tagged with a key and a case-insensitive value matching regular expressions */

Questions:

Any ideas how that would be rewritten in:
- PostgreSQL/PostGIS?
- osmdata?
Would it be worth thinking about allowing osmdata to access data stored in a PostGIS db?
Thinking: it would be great to put some of these example in vignettes / examples. Good idea?

Robinlovelace commented 7 years ago

But this works in overpass

osmcsv <- '[out:csv(::id,::type,"name")];
area[name="Bonn"]->.a;
( node(area.a)[railway=station];
  way(area.a)[railway=station];
  rel(area.a)[railway=station]; );
out;'

opq <- overpass_query(osmcsv)

mpadge commented 7 years ago

Thanks for the heads-up: This commit makes negation dead easy. This was implemented in osmplotr, but I neglected to re-insert it in Bob's overpass code. With that ability, three of the first four classes you give can be handled directly (just not !key, because that doesn't make much sense for an OSM query, and would overly burden overpass). See example at the end of ?add_feature.

regex

What osmdata was missing was any ability to do regex matches, but all that needed was changing the former exact=FALSE parameter for add_feature() to the form implemented in this commit of

add_feature <- function (opq, key, value, key_exact, value_exact, match_case, bbox)

Setting either .._exact parameter to FALSE switches to regex mode, enabling the full range of possible matching according to the overpass QL, including case-insensitivity.

That covers all your cases bar !key.

database

Would it be worth thinking about allowing osmdata to access data stored in a PostGIS db?

I'm tempted to say no only because overpass is going to implement .pbf as well as (or instead of?) .osm at some time in the future. pbf is a compressed binary form that is used for planet dumps. The minute that's done, osmdata can be made entirely compatible with any servers dishing up the OSM world. (But even then, there might not be any point plugging it into PostgreSQL, because a .pbf is a database with its own database structure that is highly efficient, so likely no gain from PostgreSQL-ing it.) Either way, wait and see would be my preference.

vignettes

Thinking: it would be great to put some of these example in vignettes / examples. Good idea?

Hell yeah. I've put a token example of negation in one line of an example, but none of the rest of this functionality is described at all, and it really ought to be. You wanna first give it a go and start assembling nice examples - and possible problems - in this issue for subsequent incorporation in the code once the reviews are in.

Robinlovelace commented 7 years ago

Thanks - I've a few more questions, please head to slack...

mpadge commented 7 years ago

All done and included in vignette with this commit

Robinlovelace commented 7 years ago

Awesome work Mark, loving the use of pipes and alternative osm APIs in that commit. osmdata is looking very good now - need to x-reference it in my stplanr-paper, but what's the citation?

mpadge commented 7 years ago

Thanks Robin! osmdata citation pending. Submitted to JOSS and should i guess be cleared promptly. I'll let you know.

Robinlovelace commented 7 years ago

One thing I picked-up on my travels: Edzer is not convinced we've adequately responded to all his comments - one to think about...

mpadge commented 7 years ago

Yeah, I'm not surprised because we haven't really actually done much except for open the issues. I was going through them the other day and realised we still actually need to address those by making the actual changes. I will get on to it asap after Brussels

Robinlovelace commented 7 years ago

Great stuff. Ping me as/when needed - you'll see I've got a substantial amount of proverbial junk to sort out in my own metaphorical castles made of sand in the form of stplanr and geocompr repos in urgent need of attention!

mpadge commented 7 years ago

@Robinlovelace @maelle http://joss.theoj.org/papers/0f59fb7eaeb2004ea510d38c00051dd3 - true to their word, the whole process took around 30 minutes. Now that's publishing!

maelle commented 7 years ago

Awesome, but it also depended on having the reviews already in via rOpenSci, sometimes you see submissions blocked quite a while in JOSS. :wink:

mpadge commented 7 years ago

Yeah, it obviously all depended on Karthik having done his homework, but even given that it was so impressively smooth. And their bot is brilliantly programmed - made it so much easier!

Robinlovelace commented 7 years ago

@mpadge any update on the promised overpass support of .pbf format?

Context: the CyIPT project I'm working on with @mem48 (who you may meet in November).

ropensci / osmdata