maproulette / maproulette-backend

MapRoulette back-end / API
Apache License 2.0
49 stars 37 forks source link

Read timeout to overpass-api after 120000 ms #889

Open Binnette opened 3 years ago

Binnette commented 3 years ago

Hello,

I love to create challenges with Overpass API, but recently I 'saw' that the default timeout was lowered from 180s to 120s 😭

So I read the source code and I saw that if my overpass query start with [out:json] or [timeout:_TIME_] then Maproulette should take in account my 'custom timeout'. But sadly something is not working properly and my custom timeout is not used.

Here is a way to reproduce the issue:

  1. Create a challenge with the following overpass query:

    [timeout:180];
    node[amenity=drinking_water];
    out;
    // DO NOT USE THIS QUERY FOR REGULAR CHALLENGE
    // Because It queries all the drinking water node worldwide!!!

    or the same query but starting with [out:json][timeout:180]

  2. Run the challenge

  3. Wait 120s to get the following error message Read timeout to overpass-api.de/xxx.xxx.xxx.xxx:80 after 120000 ms

  4. BUG! The query should have stoped at the correct 180s timeout and not an arbitrary 120s.

Part of the code to handle this timeout is in this file: ChallengeProvider.scala

Thanks all contributors for the good work done on Maproulette ❤️

hfs commented 2 years ago

Sounds like the same issue as #842 again?

Penegal commented 2 years ago

Hello, there! I got the same problem, with Overpass API request failing after 120s, when I set the request with a 1200s timeout.

For the record, the request:

[timeout:1200];
area[name="France"]->.searchArea;
(
  wr[landuse=forest][name~"omaniale"](area.searchArea);
  wr[landuse=forest][name~"ommunale"](area.searchArea);
  wr[landuse=forest][operator~"ONF"](area.searchArea);
  wr[landuse=forest][operator~"ffice"](area.searchArea);
);
out geom;
rurseekatze commented 10 months ago

Any news on this issue? I have the same problem, also tried the [timeout:1200]; statement with different values without success.

ljdelight commented 10 months ago

The request is timing out because it hits the global socket idle timeout (play.ws.timeout.idle) of 2 minutes and unfortunately WSClient does not provide a way to change the idle timeout for a specific request. So while the code does parse the (request) timeout from the overpass query and sets it on the WSClient via withRequestTimeout, the idle timeout is unchanged and the request will terminate at 2 minutes if there is no tcp activity.

Changing the global idle timeout is an option, however, setting it too high could risk system resources being tied up in idle connections. An alternative solution, in the maproulette backend source, could be to create/inject a different instance of a WSClient with a longer idle timeout and use that instance (not the globally play-injected client) for the overpass requests.

A workaround is to execute the query through the Overpass UI and then use the generated GeoJSON data to create the challenge.