onthegomap / planetiler

Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast
Apache License 2.0
1.21k stars 101 forks source link

[BUG] IOException/GOAWAY happening in parseResults() #817

Closed phanecak-maptiler closed 4 months ago

phanecak-maptiler commented 4 months ago

Describe the bug

While generating tiles with --fetch-wikidata following was observed:

...
Caused by: java.io.IOException: closed
    at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:464)
    at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:508)
    at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)
    at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
    at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:258)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:2102)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:2010)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1798)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1081)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:536)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:100)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:25)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4867)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3199)
    at com.onthegomap.planetiler.util.Wikidata.parseResults(Wikidata.java:92)
    at com.onthegomap.planetiler.util.Wikidata.queryWikidata(Wikidata.java:379)
    at com.onthegomap.planetiler.util.Wikidata.flush(Wikidata.java:314)
    ... 9 more
Caused by: java.io.IOException: /<IP of our server>:33120: GOAWAY received
    at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:1166)
...

Error was observed several times roughly a week ago, but then never again so far.

To Reproduce

While not able to reproduce the issue with WikiData server(s) and not willing to pound on them too much I've tried a mock:

  1. Setup a dummy nginx powered WikiData API mock with dummy JSON response implemented (as static file but allowing POST), HTTP/2 enabled and http2_max_requests 25; setting
  2. Adjust Planetiler to use a dummy nginx mock server
  3. Run Planetiler with --fetch-wikidata
  4. See the error

Main problem is, that I failed to trigger the error within parseResults() since I'm not sure about what were the conditions on servers:

Expected behavior

Planetiler is handling all IOException by retrying the requests, that is expected behavior.

Screenshots

N/A

Environment (please complete the following information):

Additional context

Searching around I found this: https://nuvalence.io/insights/lessons-learned-investigating-goaways-in-grpc-apis/ which may implicate http2_max_requests setting as a factor in this. Based on that I've tried to trigger GOAWAY with nginx. Also query.wikidata.org seems to be running on or behind nginx.