[BUG] API is not responsive for quite some time (at the start of the day)

elwinschmitz commented 1 month ago

Who notified us about the bug?

The 121-Platform DevOps-team (i.e. @elwinschmitz )

Describe the bug

The API (specifically the /121-program-endpoint) is not responsive (for more then 30s at least) on the first (few) requests of the day.

After some time, the API becomes "normally responsive" again. (As if the service has rebooted/properly started up by then)

To Reproduce

Steps to reproduce the behavior:

At the start of the day (around 9)
Go to https://kobo-connect-dev.azurewebsites.net/docs#/default/create_121_program_from_kobo_121_program_get
Fill in the correct, required details
See a spinner for quite some time

Expected behavior

The API should respond in a reasonable amount of time on every request. (1-10 seconds?)

Additional context

We would like to use the API in our automated E2E-tests (again), to verify our platform keeps on working as expected. So this will mean the 121-service would be making several API-requests on every PR in our repository. This will happen throughout the day/night by automated processes/bots. See https://github.com/global-121/121-platform/pull/5807

jmargutt commented 1 month ago

OK here's what I've done:

increased the number of instances that are always ready and warm for this web app from 1 to 2
enabled health check:

Health check increases your application's availability by removing unhealthy instances from the load balancer. If your instance remains unhealthy, it will be replaced.

@elwinschmitz I will leave the issue open for now, if you experience this bug again pls report here, otherwise I'll close this issue in a week's time.

jmargutt commented 1 month ago

okkidokki considering this solved

rodekruis / kobo-connect