Delayed Response Times

[cwburggrabe]

On 11/19/2024, our company who utilizes the FEMA Open API, noticed a severe delay in the response time to "https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries". I ran a few tests to help determine what the cause/solution might be, but I haven't been able to figure it out. Using this URL in PostMan results in the response returning immediately.

However, if the default Headers are removed from the request the response is delayed.

Running via a .NET results in the same about of time:

HttpClient httpClient = new HttpClient();

var requestMessage = new HttpRequestMessage()
{
    RequestUri = new Uri("https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries"),
    Method = HttpMethod.Get
};

requestMessage.Headers.Add("User-Agent", "PostManRunTime/7.42.0");
requestMessage.Headers.Add("Connection", "keep-alive");
requestMessage.Headers.Add("Accept", "*/*");
requestMessage.Headers.Add("Accept-Encoding", "*");

var responseMessage = await httpClient.SendAsync(requestMessage);

Returns in about 700ms.

Returns in about 700ms.

HttpClient httpClient = new HttpClient();

var requestMessage = new HttpRequestMessage()
{
    RequestUri = new Uri("https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries"),
    Method = HttpMethod.Get
};

var responseMessage = await httpClient.SendAsync(requestMessage);

Returns in about 36000ms

I assumed that the API needed these headers, but after adding them and running the requests on the server the delay still remained.

Was there an update that would cause this delay? I noticed that there was a changelog update listed at https://www.fema.gov/about/openfema/changelog.

[john-alfaro-fema]

Hello, Thank you for reaching out. The good news is we have been able to reproduce your issue. The bad news is it's not an issue with the underlying API. OpenFEMA API is behind a CDN/Security layer that appears to be the issue. OpenFEMA API does look at the Accept and Accept-Encoding headers, and will make some decisions, but if they are not there, it will default to sending JSON uncompressed to the user. We are reaching out to our CDN to understand why there is the added 30+ seconds to respond.

Now that you have found this issue, we are going to update our documentation to require those headers for optimal performance.

Please let us know if you continue to see issues after making those changes.

[cwburggrabe]

Hey John,

Thank you for the response. When I applied those headers to our production environment, there was still a delay for some reason using the .NET code mentioned previously.

Calling the endpoint via Chrome on the server returned the results immediately which was curious to me. I ended up copying all the headers in the initial request from the chrome dev tools into the request I made and that actually was what solved our issue.

requestMessage.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0");
requestMessage.Headers.Add("Connection", "keep-alive");
requestMessage.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7");
requestMessage.Headers.Add("Accept-Encoding", "gzip, deflate, br, zstd");
requestMessage.Headers.Add("Accept-Language", "en-US,en;q=0.9");
requestMessage.Headers.Add("Cache-Control", "max-age=0");
requestMessage.Headers.Add("Priority", "u=0, i");
requestMessage.Headers.Add("Sec-ch-Ua", "\"Microsoft Edge\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"");
requestMessage.Headers.Add("Sec-ch-Ua-Mobile", "u=0, i");
requestMessage.Headers.Add("Sec-ch-Ua-Platform", "\"Windows\"");
requestMessage.Headers.Add("Sec-Fetch-Dest", "document");
requestMessage.Headers.Add("Sec-Fetch-Mode", "navigate");
requestMessage.Headers.Add("Sec-Fetch-Site", "none");
requestMessage.Headers.Add("Sec-Fetch-User", "?1");
requestMessage.Headers.Add("Upgrade-Insecure-Requests", "1");

Quite the leap in headers but it works!