504 GATEWAY_TIMEOUT

[beckyconning]

I'm experiencing this reliably at one request often between 1052M and 1613M into the events dataset.

I could be wrong but my suspicion is that the API is just overstepping the timeout and that it occurs at this point in the dataset due to luck (the more requests in the load the more likely this is to happen).

Perhaps we can double the gateway timeout?

Thanks!

Second to last request:

Request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /drug/event.json?limit=1000&skip=0&search_after=0%3Dsafetyreport%252311937189 HTTP/1.1
Content-Length: 0
Content-Type: text/plain; charset=UTF-8
Authorization: Basic <REDACTED>
host: api.fda.gov
accept: */*
user-agent: AHC/2.1

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Server: openresty
Date: Wed, 01 Jul 2020 13:14:22 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 24446474
Connection: keep-alive
Vary: Accept-Encoding
X-RateLimit-Limit: 240
X-RateLimit-Remaining: 188
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
cache-control: no-cache, no-store, must-revalidate
Content-Security-Policy: default-src 'none'
ETag: W/"175060a-qkC43FJmAJyq/aMvX1MoimktvIg"
Link: <https://api.fda.gov/drug/event.json?limit=1000&skip=0&search_after=0%3Dsafetyreport%252311938200>; rel="next"
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Age: 2
Via: http/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Cache: MISS
Strict-Transport-Security: max-age=31536000;
Strict-Transport-Security: max-age=31536000; preload

Last request:

Request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /drug/event.json?limit=1000&skip=0&search_after=0%3Dsafetyreport%252311938200 HTTP/1.1
Content-Length: 0
Content-Type: text/plain; charset=UTF-8
Authorization: Basic <REDACTED>
host: api.fda.gov
accept: */*
user-agent: AHC/2.1

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 504 GATEWAY_TIMEOUT
Server: openresty
Date: Wed, 01 Jul 2020 13:14:27 GMT
Content-Length: 0
Connection: keep-alive
X-RateLimit-Limit: 240
X-RateLimit-Remaining: 187
Age: 5
Via: http/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Cache: MISS
Strict-Transport-Security: max-age=31536000;
Strict-Transport-Security: max-age=31536000; preload

[dkrylovsb]

The last request never reached our backend, which likely points to a network issue between the API gateway and the backend, given the fact that 504 is coming from the API gateway.

We will attempt to pinpoint the root case by working together with the API gateway team. In the meantime I would recommend that you implement a re-try strategy on your end, since the issue is intermittent and thus retrying should take care of it.

I will post back once I learn more.

[beckyconning]

Thank you!

[beckyconning]

Hi, we routinely get these errors however as you mention they are transient and so the auto retry strategy we've put in place to deal with them seems to resolve this. As such I'm going to close this issue.