Caching issue

[Flybel]

During the website maintenance today, I requested a profile search through my application to see how it handles unexpected responses. The API returned 503 as expected. Approximately 5 hours later, I tried requesting the same profile search but until now, I get the same response.

However, if I omit the &region=EU parameter (which was provided in the initial request during maintenance), I do get the correct search result. But when I add back the parameter, it responds with 503 again.

Aside from that, it seems to me like negative lookups are cached for too long (at least for responses other than 404), as even if I prevent API calls during usual maintenance times, I cannot prevent others from requesting that profile.

[man90es]

Thank you for reporting these problems. They weren't as noticeable before, but I had to increase the cache TTL on the main website from 2 to 24 hours to avoid getting IP banned again 🙄

Fixing them would require rewriting the caching mechanism to

• Store cached responses with status 503 for less time than other responses, while still caching them to avoid sending too much requests to BDO website. I don't want to hardcode wednesdays as maintenance day because I remember at least one time they moved it to another day
• Cache data based on its content rather than formatting of request string it receives

I will try to think of a way to fix them as soon as I can. In the meantime, you can self-host the API with a reduced cache TTL, either as a binary or a Docker container

[Flybel]

Thank you for your reply. I agree with not trying to work around maintenance times. For accessing the cached data, perhaps you could simply add the default parameters to the request string?

I also thought about self-hosting, but decided against it because my application runs on a very constrained VPS. However, I was not aware that your instance caches data for that long, my application will need to fetch one or few guild profiles often, so I will look into that again.

[man90es]

Constrained VPS shouldn't be an issue. I use the cheapest one I could find

[Flybel]

Thank you, I will definitely make my own instance then.

By the way, I have noticed in ScrapeAdventurer.go that if the guild element is of type span, it is always treated as private, even though on the website it can occur as "Private" or "Not in a guild". Accordingly, even though my app distinguishes these cases by checking if the privacy bitflag is set when guild is undefined, the result is always Private. I saw that other fields such as contribution points check for the element's text, could this be applied to the guild field as well?

[man90es]

Thanks for reporting that too. They used to have an \<a> tag leading to nowhere for players not in a guild before

[man90es]

Version 1.5 with fixes for all described in this thread bugs is out. Please open a new issue if they somehow reappear or if you spot any new ones