Frequent 504s from Metaforecast API

[mqp]

I'm working on the pipeline that Base Rate Times uses to get its data from Metaforecast, and it's difficult to get reliable responses from the Metaforecast API. For example, this query frequently fails with a 504:

{
  question(id: "goodjudgmentopen-2617") {
    history {
      options {
        name
        probability
      }
      fetchedStr
    }
  }
}

{
  "error": {
    "code": "504",
    "message": "An error occurred with your deployment"
  }
}

When it does work, it's also very slow.

Doing concurrent requests (e.g. for multiple markets at once) seems to make it more likely to timeout and fail.

Making this more reliable would make our lives a lot easier! Appreciate the service.

[NunoSempere]

Received.

Are you also getting problems when just fetching the latest probability? Is there some way you could just fetch the latest probability, rather than the full history every time?

[NunoSempere]

E.g., using this query instead is almost instant and should work well:

{
  question(id: "goodjudgmentopen-2617") {
      options {
        name
        probability
      }
      fetchedStr

  }
}

[mqp]

Interesting. The reason is because Base Rate Times doesn't have a database of its own, it just runs some scripts to scrape the full history for its charts from Metaforecast every time it wants updated numbers for those charts.

I don't think there's a way in your GraphQL endpoint to query for a timestamp > some previous checkpoint, right? So even if we had a database with the existing data, I am not sure we could use this technique to get what we were missing and compile an up-to-date chart.

[berekuk]

This looks like a problem with missing DB indexes, I'll try to fix it in a few hours.

[mqp]

That makes sense, thank you very much!

[berekuk]

Should be much better now!

[mqp]

Confirmed that the latency is under control now -- it takes only a few hundred ms for queries of this form for different markets. (By the way, I didn't mention this, but there was a large caching effect before, where running the same query twice would be fast the second time.)

Thanks for the fix!