search

gluonhq / maps

https://gluonhq.com/labs/maps/
GNU General Public License v3.0
140 stars 36 forks source link

OSM returns 403 for http, 429 for https #33

Closed jperedadnr closed 4 years ago

jperedadnr commented 5 years ago

Recently OpenStreetMap rejects the request to retrieve tiles. For http://tile.openstreetmap.org/, it returns on every request:

java.io.IOException: Server returned HTTP response code: 403 for URL: http://tile.openstreetmap.org/5/19/10.png
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1913)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
        at com.gluonhq.maps/com.gluonhq.impl.maps.tile.osm.CachedOsmTileRetriever$CacheThread.doCache(CachedOsmTileRetriever.java:190)
        at com.gluonhq.maps/com.gluonhq.impl.maps.tile.osm.CachedOsmTileRetriever$CacheThread.run(CachedOsmTileRetriever.java:157)

Changing to https doesn't solve the issue, though, as it rejects it now with a different error:

java.io.IOException: Server returned HTTP response code: 429 for URL: https://tile.openstreetmap.org/6/37/22.png
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1913)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:245)
        at com.gluonhq.maps/com.gluonhq.impl.maps.tile.osm.CachedOsmTileRetriever$CacheThread.doCache(CachedOsmTileRetriever.java:190)
        at com.gluonhq.maps/com.gluonhq.impl.maps.tile.osm.CachedOsmTileRetriever$CacheThread.run(CachedOsmTileRetriever.java:157)

As the URL (http or https) works fine on a browser, it seems that adding user-agent to the URLConnection request could solve it. However, the tiles return blank.

A working solution for Desktop is this: setting http.agent system property:

static {
        System.setProperty("http.agent", "Mozilla/5.0");
    }

For Android:

static {
    String userAgent = System.getProperty("http.agent");
    System.setProperty("http.agent", userAgent);
}

Related StackOverflow question: https://stackoverflow.com/questions/56397206/gluon-maps-doesnt-load-the-map-and-throws-an-exception

scaidermern commented 5 years ago

Sorry, but faking the user-agent is a bad idea. This app seems to violate OSM's tile usage policy. OSM servers run entirely on donated resources. Please set a valid user-agent pointing to Gluon Maps, not to Mozilla.

jperedadnr commented 5 years ago

Hi @scaidermern, thanks for reporting.

We don't intend to violate any policy of course. We just found out about the issue with the http response errors a few days ago, and while we don't have a definitive fix yet, we are just trying some workarounds.

When running on Desktop there is no http.agent defined, while on Android default agent, which is already is giving something like:

Dalvik/2.1.0 (Linux; U; Android 9; Pixel XL Build/PQ3A.190505.001)

based on the user's device.

So we can set for instance:

    static {
        String httpAgent = System.getProperty("http.agent");
        if (httpAgent == null) {
            httpAgent = "(" + System.getProperty("os.name") + " / " + System.getProperty("os.version") + " / " + System.getProperty("os.arch") + ")";
        }
        System.setProperty("http.agent", "Gluon Mobile/1.0.3 " + httpAgent);
}

Would that work?

. Of course, we can add to that:

static {
        System.setProperty("http.agent", "Gluon Mobile/1.0.3  /" + System.getProperty("http.agent"));
}
scaidermern commented 5 years ago

Yes, looks fine as long as the string "Gluon Mobile" will be included in the user-agent.

kazshimda commented 5 years ago

Dear Jose Perada I'm very happy to find your issue. I'm in trouble because I cannot display OpenStreetMap in MATSim. MATSim is a traffic simulation written in Java. There is a description that reads OSM file in TileFactoryBuilder class in MATSim. I would be grateful if you coud give me advice on how to add the your proposal method that add http.agent. The contents of TileFactoryBuilder.class are described below.

kazshimda commented 5 years ago

/* *** *

package org.matsim.evacuationgui.jxmapviewerhelper;

import org.jdesktop.swingx.mapviewer.DefaultTileFactory; import org.jdesktop.swingx.mapviewer.TileFactory; import org.jdesktop.swingx.mapviewer.TileFactoryInfo; import org.jdesktop.swingx.mapviewer.wms.WMSService;

public abstract class TileFactoryBuilder {

public static TileFactory getOsmTileFactory() {

final int max=17;
    TileFactoryInfo info = new TileFactoryInfo(0, 17, 17,
            256, true, true,
"http://tile.openstreetmap.org",
            "x","y","z") {           

        @Override
        public String getTileUrl(int x, int y, int zoom) {
            zoom = max-zoom;
            String url = this.baseURL +"/"+zoom+"/"+x+"/"+y+".png";
            return url;
        }

    };
    TileFactory tf = new DefaultTileFactory(info);
    return tf;
}

public static TileFactory getWMSTileFactory(String baseURL, String layer) {

    WMSService service = new WMSService(baseURL, layer);

    return new WMSTileFactory(service,17);

}

private static final class WMSTileFactory extends DefaultTileFactory {
    public WMSTileFactory(final WMSService wms, final int maxZoom) {
        super(new TileFactoryInfo(0, maxZoom, maxZoom, 
                256, true, true, // tile size and x/y orientation is r2l & t2b
                "","x","y","zoom") {
            @Override
            public String getTileUrl(int x, int y, int zoom) {
                int zz = maxZoom - zoom;
                int z = (int)Math.pow(2,(double)zz-1);
                return wms.toWMSURL(x-z, z-1-y, zz, getTileSize(zoom));
            }

        });
    }
}

}

jperedadnr commented 4 years ago

@kazshimda Did you see the changes in this commit https://github.com/jperedadnr/maps/commit/1471ef434e5bbf2feda4de656b99f739e542a770 ? You could add something like that, replacing Gluon Maps with your MatSim or alike.

kazshimda commented 4 years ago

Thank you for your quick reply. The place where I display OSM is not in the web browser, but in the Graphical User Interface of MATSim. I'm looking for a setting of user-agent notation instead of Gluon Maps that you use.

On the other hand, the OSM tile web site may change around June this year, which has changed the way to get OSM tile from java programs. Do you know if there was any announcement from the OSM administrator? Or Do you know which OSM representative to contact about this issue?

jperedadnr commented 4 years ago

Gluon Maps doesn't use a web browser either. We simply add that commit I linked before to the Java code.

By the way, have you tried using https instead of http (as in https://tile.openstreetmap.org)? Probably this will solve your issues. I recall this being the main change from a few months ago that you also mentioned.

kazshimda commented 4 years ago

@jperedadnr Thank you for your reply I changed to https and run MATSim program, but OSM was not displayed in MATSim's GUI.

The error message of MATSim is described below. I'm thinking about and trying which class of program and where in the program to add about http.agent like you. It would be helpful if you could give me some advice.

Failed to load a tile at url: http://tile.openstreetmap.org/15/8635/12957.png
java.io.IOException: Server returned HTTP response code: 403 for URL: http://tile.openstreetmap.org/15/8635/12957.png
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.net.URL.openStream(Unknown Source)
    at org.jdesktop.swingx.mapviewer.AbstractTileFactory$TileRunner.cacheInputStream(AbstractTileFactory.java:348)
    at org.jdesktop.swingx.mapviewer.AbstractTileFactory$TileRunner.run(AbstractTileFactory.java:307)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
jperedadnr commented 4 years ago

It doesn't look like you applied https, it shows you are trying to load from http. I've already mentioned the fix we have done in Gluon Maps, that still uses http. With https it shouldn't even be necessary.

kazshimda commented 4 years ago

@jperedadnr Sorry the error message I posted earlier was an error in http status. The error when changing to https is described below. The error content looks almost same without at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source).

Failed to load a tile at url: https://tile.openstreetmap.org/15/8635/12956.png
java.io.IOException: Server returned HTTP response code: 429 for URL: https://tile.openstreetmap.org/15/8635/12956.png
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
    at java.net.URL.openStream(Unknown Source)
    at org.jdesktop.swingx.mapviewer.AbstractTileFactory$TileRunner.cacheInputStream(AbstractTileFactory.java:348)
    at org.jdesktop.swingx.mapviewer.AbstractTileFactory$TileRunner.run(AbstractTileFactory.java:307)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
tiainen commented 4 years ago

Error code 429 means that you are blocked most likely because of a rate limit that has been exceeded. This happens when you try to download too many tiles in a short period of time. See https://operations.osmfoundation.org/policies/tiles/ for the tile usage policy.

kazshimda commented 4 years ago

@tiainen Thank you for telling me what my error is. Error number 429 indicates that there is a lot of data, so I tried using small OSM data. But again, the same 429 error was displayed. From this result, I think that the size of the map imported from OSM is not the cause. I will continue the investigation. Thank you all.

abhinayagarwal commented 4 years ago

Hi @kazshimda,

I will close this issue now. If the issue still persists, please free to reopen.