search

Neos-Metaverse / NeosPublic

A public issue/wiki only repository for the NeosVR project
195 stars 9 forks source link

Neos Backend Services Latency and Rate Limit Issues #3914

Open stiefeljackal opened 1 year ago

stiefeljackal commented 1 year ago

Describe the bug?

Based on my logs from June 7th to June 11th as well as the rise of syncing issues, it appears that the backend services from api.neos.com (as well as from cloudx.azurewebsites.net) are experiencing issues where it will respond with 429 Too Many Requests errors. As of now, it appears the following areas within the service are being affected:

Relevant issues

None reported.

To Reproduce

Sessions and Friends 429 Errors

Startup your client, wait, and open your logs for the current session. You should see Server Error messages for the above mentioned endpoints (/api/sessions and /api/users/U-MyAccount/friends). Within the log, open any of the URLs in your browser and perform a regression testing of performing a handful of GET requests against those URLs. From the testing, you should notice latency of the response and the occasional 429 error.

Record Sync

Hold a world build session and start saving imported assets in your inventory. From this, a stuck sync should occur. This happened during CJ 211.

Unfortunately, these are the best steps to reproduce the issue since this is mainly a backend issue and not a client issue. Logs from the backend services will need to be reviewed by the development team to see what is going on.

Expected behavior

Log Files

JACKAL_MK2 - 2022.1.28.1310 - 2023-06-10 23_49_32.zip

Pay attention to Server Error running GET request to... messages in logs.

Screenshots

No response

How often does it happen?

Always

Does the bug persist after restarting Neos?

Yes

Neos Version Number

2022.1.28.13XX

What Platforms does this occur on?

Windows, Linux

Link to Reproduction Item/World

No response

Did this work before?

Yes

If it worked before, on which build?

2022.1.28.13XX

Additional context

Others may add to this report as this problem is slowly starting to become a real issue.

Reporters

U-StiefelJackal (Discord: Stiefel Jackal | stifeljackal)

Will include others when more findings are submitted.

ginjake commented 1 year ago

I had a similar situation myself. I tried the following without success and was having trouble Delete the mod Restart Neos (steam version) Start Neos (StandAlone version) Reboot PC Delete Cache and database.

Finally, I rebooted my Wifi router at home and it worked!

I don't understand English, so I use deepl to translate!

stiefeljackal commented 1 year ago

I had a similar situation myself. I tried the following without success and was having trouble Delete the mod Restart Neos (steam version) Start Neos (StandAlone version) Reboot PC Delete Cache and database.

Finally, I rebooted my Wifi router at home and it worked!

I don't understand English, so I use deepl to translate!

This is something that is on the backend services side (api.neos.com and others) and not a local issue. Mainly, the issue is with CosmosDB when the time this issue was submitted. @bontebok from QC is monitoring the situation, but it appears the issue is not too bad at the moment. You still get the occasional 429 or 404 from CosmosDB still.

kazu0617 commented 1 year ago

I think this problem based RU/s setting in azure, it written in Microsoft Document. https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-request-rate-too-large?tabs=resource-specific

stiefeljackal commented 1 year ago

I think this problem based RU/s setting in azure, it written in Microsoft Document. https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-request-rate-too-large?tabs=resource-specific

That’s correct. Please see the QC channel in the Neos Volunteer Discord server for the full history as @bontebok and I are monitoring this to the best of our abilities.