Foundations API - Receiving TooManyRequests Responses (429) after Gateway Timeout Responses (504)

[BYM-Henry]

Hi Reapit Foundations Team,

An issue seems to have returned, which we raised a couple of years ago:

4414

Describe the bug When performing a full pull of data for a client using our application at a rate of 5 concurrent requests and 10 requests per second, after receiving a 504 response, for a period of time all subsequent requests receive a 429 response.

From the behaviour it seems like whatever is timing out the request on your side is not communicating correctly with the rate limiter to tell it that the request has finished.

To Reproduce Have 5 concurrent requests running. Have 1 of them return a 504 response. Make another request and receive a 429 response.

Expected behaviour We would expect for a request to not receive a 429 response as long as we stay in the concurrency and request per second limits.

Let me know if you have anymore questions.

Many Thanks,

Henry Fitch BriefYourMarket

Specification

• Project: Reapit.Packages.RequestConcurrency
• This should already be being handled correctly, however for serverless services, it could be that the Lambda timeout is occurring before the query time out on the database, which means that the code to release the semaphore lock in the package above never actually gets executed.
• Confirm which endpoints are causing the problem (can use logs for this). If it's serverless APIs, the point above likely applies in which case increasing the function timeout will probably resolve the problem.
• Check query timeout in the APIs (these should be 29 seconds)

To setup dev environment you'll need to be running Redis locally (or inside a WSL linux container on your machine) and run any core API (ie Reapit.Services.Contacts) with a configuration that points your local Redis instance. Issuing requests will allow you to observe the semaphore action happening in the package to manage the number of concurrency requests allowed. What we're looking for here is situations where a lock can be obtained that's never released

[github-actions[bot]]

Thank you for taking the time to report a bug. We prioritise bugs depending on the severity and implications, so please ensure that you have provided as much information as possible. If you haven’t already, it really helps us to investigate the bug you have reported if you provide ‘Steps to Replicate’ and any associated screenshots. Please ensure any personal information from the production database is obscured when submitting screenshots. This issue will be reviewed in our weekly refinement sessions and assigned to a specific project board. We may also update the ticket to request additional information, if required. For more information on our processes, please click here

[plittlewood-rpt]

@BYM-Henry please can you just confirm which endpoints you're having trouble with? I've got a hunch as to what this could be.

[github-actions[bot]]

This issue has been updated and moved to our ‘Near Term’ column (typically completed within 0 - 4 months). We have assessed the effort required and outlined a technical specification - please take the time to review this detail. When we're ready to schedule the issue, it will be assigned to the relevant board where you can continue to track its progress to completion. For more information on our processes, please click here

[BYM-Henry]

Hi @plittlewood-rpt

The endpoint that is causing us the most timeouts currently (for customer YOM), is the Appointments endpoints. This is using the changes suggested a couple of weeks ago (no filter or sort on the request). However in regards to the subsequent 429 responses, this is affecting all endpoints (that we use) as we are hitting them concurrently.

We can generally handle a few timeouts as we use an exponential backoff and normally these requests will succeed later on in the day/week. But its causing us issues as the timeouts are then resulting in requests that would normally succeed to fail.

Cheers,

Henry

[cduggan-reapit]

Hi @BYM-Henry,

We have adjusted the timeout settings, which should give the services sufficient time to release the concurrency locks when internal connections time out. Please let us know if you continue to receive 504 responses.

[github-actions[bot]]

It looks like you have commented on a closed issue. If your comment relates to a bug or feature request, please open a new issue, and include this issue number/url for reference. For more information on our processes, please click here