Imagery blacklist blocks Google Cloud Storage hosted imagery

[ramSeraph]

The current imagery blacklist meant to block google maps imagery also blocks imagery hosted on Google cloud storage.

Current regex at https://github.com/openstreetmap/openstreetmap-website/blob/master/config/settings.yml#L89

sample google cloud storage url: "https://storage.googleapis.com/bucket_name/obj_name"

[pnorman]

What is an example URL?

[ramSeraph]

Updated in the issue description

[ramSeraph]

a working example here - https://storage.googleapis.com/soi_data/export/tiles/{z}/{x}/{y}.webp

[ramSeraph]

I don't mind submitting a PR to fix this, if there is confirmation that it will be let through.. i.e. if there is no ideological objections to this

[hlfan]

If there's no imagery whitelist and only the blacklist regex to update, then the solution in #3512 should do it.

[ramSeraph]

as #3512 is a closed PR.. i am going to ask here.. what is stopping the closure of this issue.. is it a design decision on whether a white list needs to be introduced because of lack of complex regex support in some browsers or is it just a lack of someone to create a PR?

If it is the later case... I will submit the PR which ever way you decide to go

[ramSeraph]

Or is there is there a third direction where you plan to drop the whole blacklist thing, because it is clearly decorative and can be overcome by setting up a simple redirection server? (which is what I am currently doing anyway)

[gravitystorm]

Or is there is there a third direction where you plan to drop the whole blacklist thing, because it is clearly decorative and can be overcome by setting up a simple redirection server? (which is what I am currently doing anyway)

No, because this blacklist is not to strictly enforce a policy, but to prevent people who don't know about the legality of sources from making mistakes. There's a lot of instances where people don't know they aren't supposed to do something, or who don't care and will do it if it's easy, and a much, much smaller number of cases of people who are determined (and skilled) enough to do things like set up a redirection server.

As for the regexps, I haven't yet had a chance to investigate any of the implications mentioned above, so I can't answer which direction for a solution that I would prefer, yet.

[ramSeraph]

Then your blacklist should be much much longer... anyway... not going to argue any further. Please do see what you can do to move the issue forward. Thanks.

[ramSeraph]

I was told I should tell you the importance of the map you are blocking and why you should prioritise this.. this map happens to be the only open map we ever got from SOI. And the redirect i made was a temporary solution and I don't intend to maintain it anymore because I believe in only maintaining things that are really required. So basically you will be blocking a lot of OSM India work by not prioritising this

[grischard]

Then your blacklist should be much much longer... anyway... not going to argue any further. Please do see what you can do to move the issue forward. Thanks.

@ramSeraph your tone with the volunteers who maintain the website isn't appropriate.

The best solution for OSM India is to host the imagery elsewhere. How large is the data? Maybe the OSMF can host it.

[ramSeraph]

I also happen to be a volunteer who collected the map and been hosting it at my own expense and making it available to OSM India. But, I don't believe that what I said was inappropriate but I do apologize if it hurt the feelings of the people involved.

But blacklisting a whole domain because an appropriate regex couldn't be found and then not correcting the problem after pointing it out a month back that a valid use case was being blocked was not ok either.. I did volunteer to fix the regex. All I needed was a decision. And fixing a regex should be an easier solution than moving data around.

[ramSeraph]

If the regexes are that hard to maintain, maybe it would be a good idea to drop them.

[pnorman]

When writing the original regular expressions, I intentionally avoided negative lookbehinds or any other implementation-specific features, because the regular expressions need to work over an assortment of browsers, Java, and whatever else someone might implement an editor in.

[ramSeraph]

So, if a solution can't be found with a non negative lookbehind, a whitelist would be a better idea? Would that mean a API version change? If a whitelist is introduced without an API version change is there a chance of breaking any clients?

[ramSeraph]

I am mostly concerned with JOSM.. if a whitelist is introduced.. I will go and make the necessary changes in JOSM

[gravitystorm]

As for the regexps, I haven't yet had a chance to investigate any of the implications mentioned above, so I can't answer which direction for a solution that I would prefer, yet.

OK, I've had a look through the related issues and thanks @pnorman for the additional information.

I don't have a preferred solution, since I'm not a regexp expert. So I'll outline the constraints, and then I'm happy to review any PR that meets those constraints. But please, realise that there's never a guarantee that any specific PR will be accepted, so please no complaining if a proposed PR is declined and also please no complaining if it takes a while for any PR to be reviewed.

• I'm happy to allow user-hosted imagery on google cloud storage. The purpose of the blacklist isn't to prevent this.
• The regular expressions should not rely on "negative lookbehinds" or other features that are not cross-browser or not well supported in most programming languages. I'll defer to other people's knowledge on this topic, if required. This is because our API is implementation-agnostic so "it works in JOSM" or "it works in iD in some browsers" is not sufficient.
• The PR must come with tests for the regular expressions demonstrating sample urls that should be blocked and sample urls that should be permitted.
• There must be tests to show that it continues to block google map images and google aerial imagery.
• There must be tests to show that user-hosted imagery on google cloud storage is permitted.
• There must be some narrative explanation as to why google are unlikely to use google cloud storage for their own maps and/or imagery, either now or in the near term, since I don't want to end up having some kind of per-bucket blacklist or whitelist if they start hosting their own maps/imagery on the storage.googleapis.com domain too.

[mmd-osm]

Is this still relevant? I've found an issue where OP referred to some other URL where the data has moved to https://github.com/ramSeraph/opendata/issues/22

[mmd-osm]

Closing now due to lack of feedback. Please ping again in case we should reopen this issue.