Sophox queries fail

[matkoniecz]

./gradlew updateAtmOperators is supposed to update country data about operator tags on ATM. I rerun it as I fixed many of such problems in Poland and a bit in other areas (manual script-assisted normalization, say operator=EURONET, operator=euronet to operator=Euronet).

Sadly this script is not working anymore, it appears to suffer from some kind of failure. Typically it results in producing new version with most of entries removed, despite that many of removed entries actually should be listed.

Running it manually sometimes seems to mostly work, sometimes it goes down with java.util.concurrent.ExecutionException: org.openrdf.query.MalformedQueryException: com.bigdata.rdf.sail.sparql.ast.VisitorException: QName 'osmt:amenity' uses an undefined prefix.

I have not investigated whether returned data is complete. Maybe some format of something changed? Maybe Sophox instance is again not working?

For reference, one of queries is

SELECT ?value ?loc
WHERE { ?osm osmt:amenity 'atm'; osmt:operator ?value; osmm:loc ?loc. }

and can be run manually on sophox.org. See https://sophox.org/#SELECT%20%3Fvalue%20%3Floc%0AWHERE%20%7B%20%3Fosm%20osmt%3Aamenity%20%27atm%27%3B%20osmt%3Aoperator%20%3Fvalue%3B%20osmm%3Aloc%20%3Floc.%20%7D

I asked in https://wiki.openstreetmap.org/wiki/Talk:Sophox#Tag%20usage%20statistics in case that there is some obvious solution.

[westnordost]

why do you post this here? The Exception is not from the script, but from sophox

[matkoniecz]

Because script used in StreetComplete is broken and should be either changed to fit whatever Sophox is doing or replaced somehow.

Yes, it become broken due to third-part tool changing/breaking down but nevertheless it is broken :(

[westnordost]

Actually, I can't reproduce. The script works just fine.

However, what still doesn't work fine is that Sophox only returns a fraction of the actual OSM data. This is why the three sophox dependent tasks are commented out in updateStreetCompleteData. The sophox service itself doesn't work properly currently. The people running the service already know about it, nothing more we can do here.

[westnordost]

See also https://github.com/Sophox/sophox/issues/27

[matkoniecz]

But we still need to replace it by something working, right?

I have following ideas:

• ignore it (but I really want to fix autocomplete in Poland - several variants of Euronet are irritating)
• wait for Sophox to be fixed (seems very unlikely, it is broken since November 2020 - first full outage, later missing most of OSM data - https://github.com/Sophox/sophox/issues/24 )
• fix Sophox (not planning to do this and have no access anyway)
• overpass turbo queries (extremely resource intensive, but should be fine for a yearly update)
• download full planet data and process it (similarly to how NSI is doing, maybe their code can be reused)
• fetch data from NSI (though they base it on names, not operator and with much higher thresholds)
• start updating it manually - nonononono

[westnordost]

Well, I guess overpass could be an option. But because i have otger things to do, I'll still try to sit it out, i.e. hope that one time, they'll get it fixed

[westnordost]

Can you group results by country code and count em?

[matkoniecz]

Can you group results by country code and count em?

Nope, it would be necessary to do query for each country separately or geocode results. I did both versions, so it is possible.

But it is a bit dumb way of doing things.

[westnordost]

geocoding the results should be quite swift when using my country boundaries library

[matkoniecz]

https://wiki.openstreetmap.org/wiki/Ohsome_API may be also viable, seems to be sort of similar and more stable

[matkoniecz]

1) I will update for example ATM operators in a near future

2) there is no sane replacement (I will update using bespoke overpass hack that is not suitable for running more often than once several years, ohsome seems not really fitting and spending time on improving it just to solve this problem seems poor choice of effort, sophox is well dead, processing entire planet would be also a big effort for minor gain, NSI is far from collecting all atm operators) - so I will wotfix this issue