500 Internal Server Error occurs occasionally

[josephlyu]

Hi! I have been using the api for a few weeks and it works great for most of the time, but I'm occasionally having the FailedRequestError (500 - Internal Server Error), and it seems that it still works after I retried for multiple times (like 10 times, it's still fine in a jupyter notebook, but just won't be much pleasant when integrated into the project with multiple py files). Here is my code:

filters_uk = ['areaType=overview']

structure_total = { "date": "date", "newcases": "newCasesByPublishDate", "cumcases": "cumCasesByPublishDate", "newdeaths": "newDeaths28DaysByPublishDate", "cumdeaths": "cumDeaths28DaysByPublishDate" }

api_uk = Cov19API(filters_uk, structure_total)

It is a very basic data request and it works fine till here.

The problem only occurs when I try to get time or the data.

api_uk_timestamp = api_uk.last_update

and

df_uk = api_uk.get_dataframe()

And as I descriped, both of the operations will still work anyway if I retry many times, and the problem only occurs very occasionally(I only had twice in three weeks, last time it was sovled after I restart the kernel/laptop, but this time it didn't work that way), so I am not sure whether it is only something to do with myself or it can be improved, but I'll just report it as it suggested in the error message.

Many thanks!

[bhavesh0009]

API works with one or few LTLAS but not for all LTLAS.

Example code

from uk_covid19 import Cov19API
import pandas as pd

ltla_filter = ['areaType=ltla']
cases_and_deaths = {
                    "areaType":"areaType"
                    ,"areaName":"areaName"
                    ,"areaCode":"areaCode"
                    ,"specimenDate":"date"
                    ,"dailyLabConfirmedCases":"newCasesBySpecimenDate"
                    ,"totalLabConfirmedCases":"cumCasesBySpecimenDate"
                    }
api = Cov19API(filters=ltla_filter, structure=cases_and_deaths)
data = api.get_json()  # Returns a dictionary    

RuntimeError: Request failed: {"response":"An internal error occurred whilst processing your request, please try again. If the problem persists, please report as an issue and include your request.","status_code":500,"status":"Internal Server Error"}

[xenatisch]

I have checked the servers. We've had ~15k server (5xx) errors since 4pm. That's out of 28.6 million requests.

The UK West servers are showing increase in latency, but then again, the traffic has been extremely high and still still going on. That said, UK South servers are performing well (both instances).

I'll try to make some changes to reduce server-side latency even more, but as things currently stand, there isn't much I can do I'm afraid. The traffic is still at around 90k req/min, and it's certainly not the usual pattern...

[xenatisch]

@josephlyu It should be alright again. Let me know if you're still experiencing any difficulties.

[josephlyu]

@xenatisch Thank you! Yeah it works alright now, and your explaination did answer my question (as I'm also based in the north). I'll be closing the issue.