Currently online display requests all currently online users from API, thus eventually exceeding rate limits

[bdach]

Originally posted by @bdach in https://github.com/ppy/osu/issues/29968#issuecomment-2371064573

Here's a log on this PR showing it still happening.

Well that was a bit of misdirection wasn't it, since the original snippet you posted showed the beatmap set request failing, but I'd be willing to bet money on the fact that it's all of these that cause the ratelimit to get exceeded:

2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36732905&ids[]=35262412&ids[]=35875909&ids[]=29867634&ids[]=18633882&ids[]=20822457&ids[]=33201184&ids[]=18230719&ids[]=34489118&ids[]=6082303&ids[]=9049614&ids[]=10915715&ids[]=34555317&ids[]=2605324&ids[]=34399173&ids[]=33622436&ids[]=9290272&ids[]=36725275&ids[]=30553311&ids[]=36634355&ids[]=7093548&ids[]=20084327&ids[]=33407487&ids[]=34429696&ids[]=9961743&ids[]=15280942&ids[]=34035397&ids[]=21478222&ids[]=36692537&ids[]=29876889&ids[]=36182515&ids[]=12485807&ids[]=36245822&ids[]=36660354&ids[]=8985720&ids[]=22096637&ids[]=13440678&ids[]=15595510&ids[]=19972098&ids[]=36179593&ids[]=13274097&ids[]=35197617&ids[]=3128601&ids[]=10411604&ids[]=19886894&ids[]=36724819&ids[]=11358214&ids[]=27621192&ids[]=15967940&ids[]=14471463 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 100,825 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36256780&ids[]=35607852&ids[]=21683307&ids[]=36534979&ids[]=36563271&ids[]=36715683&ids[]=10440535&ids[]=36200271&ids[]=12760514&ids[]=36598466&ids[]=35451125&ids[]=35205599&ids[]=19315458&ids[]=36536998&ids[]=34491452&ids[]=10047047&ids[]=30849975&ids[]=35075540&ids[]=33357362&ids[]=16769621&ids[]=24193502&ids[]=16884064&ids[]=15720951&ids[]=20782611&ids[]=14915346&ids[]=35024623&ids[]=35226190&ids[]=31473916&ids[]=19228035&ids[]=33309529&ids[]=24026160&ids[]=10192982&ids[]=29490192&ids[]=36559045&ids[]=35270752&ids[]=33946246&ids[]=36372274&ids[]=11139822&ids[]=7601737&ids[]=10839558&ids[]=36729023&ids[]=33590999&ids[]=17119609&ids[]=32448556&ids[]=33485239&ids[]=34849185&ids[]=6763446&ids[]=36494718&ids[]=36428479&ids[]=3371797 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 98,050 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36206865&ids[]=34891056&ids[]=15308543&ids[]=23853662&ids[]=36636445&ids[]=36722342&ids[]=34548393&ids[]=26680420&ids[]=36732629&ids[]=19188154&ids[]=33250965&ids[]=36159593&ids[]=36634268&ids[]=3581753&ids[]=34024700&ids[]=33635230&ids[]=12612018&ids[]=28163355&ids[]=33157119&ids[]=19019042&ids[]=36540414&ids[]=24548042&ids[]=36290492&ids[]=36738436&ids[]=15158331&ids[]=14558166&ids[]=36734335&ids[]=16339271&ids[]=4615046&ids[]=14130446&ids[]=25950535&ids[]=36003268&ids[]=23424019&ids[]=32477801&ids[]=1483160&ids[]=36339083&ids[]=36668541&ids[]=36692583&ids[]=35392462&ids[]=35896616&ids[]=9230310&ids[]=33991360&ids[]=11327608&ids[]=11854911&ids[]=21706431&ids[]=20660557&ids[]=33361386&ids[]=24521426&ids[]=36732826&ids[]=34586280 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 99,290 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36738179&ids[]=16215405&ids[]=36254961&ids[]=36176782&ids[]=33103283&ids[]=24778953&ids[]=11291084&ids[]=36585772&ids[]=28200630&ids[]=30023521&ids[]=34215242&ids[]=2632240&ids[]=4476818&ids[]=36478491&ids[]=33496129&ids[]=35502219&ids[]=16446112&ids[]=2503170&ids[]=20287733&ids[]=35015946&ids[]=32357147&ids[]=17821735&ids[]=15496039&ids[]=33180552&ids[]=33850381&ids[]=35637077&ids[]=35801873&ids[]=36495328&ids[]=33915524&ids[]=25054049&ids[]=32923736&ids[]=36732015&ids[]=32917135&ids[]=35831195&ids[]=36503785&ids[]=26008189&ids[]=32507269&ids[]=36694881&ids[]=7173453&ids[]=36301670&ids[]=36447635&ids[]=35429541&ids[]=17686915&ids[]=8370475&ids[]=21843822&ids[]=32322458&ids[]=29071427&ids[]=20746542&ids[]=21753748&ids[]=36155608 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 103,764 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=11912649&ids[]=36320162&ids[]=36016646&ids[]=36167473&ids[]=18813137&ids[]=34848135&ids[]=34838867&ids[]=13737273&ids[]=31325331&ids[]=3030502&ids[]=36738569&ids[]=36720321&ids[]=34032576&ids[]=36590886&ids[]=6572940&ids[]=36411962&ids[]=16896580&ids[]=30058248&ids[]=35931211&ids[]=32910101&ids[]=19010333&ids[]=34641461&ids[]=36689884&ids[]=9626734&ids[]=34074744&ids[]=25156562&ids[]=33038882&ids[]=17848299&ids[]=29944108&ids[]=15854832&ids[]=35194093&ids[]=10993158&ids[]=16597584&ids[]=33893911&ids[]=35372081&ids[]=22521306&ids[]=22123643&ids[]=17368545&ids[]=23765997&ids[]=26544222&ids[]=17269597&ids[]=35054605&ids[]=16161649&ids[]=25924476&ids[]=21165636&ids[]=34723790&ids[]=24515690&ids[]=34837309&ids[]=31065309&ids[]=7982003 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 98,377 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/images/layout/avatar-guest@2x.png successfully completed!
2024-09-23 14:52:22 [verbose]: Request to https://a.ppy.sh/33467060?1682039646.jpeg successfully completed!
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetBeatmapsRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36337052&ids[]=10730875&ids[]=36729403&ids[]=8595858&ids[]=13268182&ids[]=15037158&ids[]=25232008&ids[]=10674648&ids[]=24628787&ids[]=36393336&ids[]=24591855&ids[]=36595378&ids[]=36150562&ids[]=31402882&ids[]=36730640&ids[]=36206724&ids[]=18014083&ids[]=19331260&ids[]=17697613&ids[]=36733066&ids[]=35998860&ids[]=9519610&ids[]=31003097&ids[]=33351652&ids[]=25781802&ids[]=32563747&ids[]=34119064&ids[]=31861680&ids[]=33024696&ids[]=36738593&ids[]=19114005&ids[]=6904819&ids[]=36738536&ids[]=33819981&ids[]=34673097&ids[]=12579466&ids[]=36677444&ids[]=35998757&ids[]=35263887&ids[]=17712448&ids[]=36094734&ids[]=36734703&ids[]=12697651&ids[]=16124176&ids[]=35059914&ids[]=29405230&ids[]=35920077&ids[]=32606396&ids[]=36469769&ids[]=36271263 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 93,650 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/beatmaps/?ids[]=1219690 failed with System.Net.WebException: TooManyRequests.
2024-09-23 14:52:22 [verbose]: Response was: {"error":"Too Many Attempts."}
2024-09-23 14:52:22 [verbose]: Failing request osu.Game.Online.API.Requests.GetBeatmapsRequest (osu.Game.Online.API.APIException: Too Many Attempts.
2024-09-23 14:52:22 [verbose]: ---> System.Net.WebException: TooManyRequests
2024-09-23 14:52:22 [verbose]: --- End of inner exception stack trace ---)

To resolve this we'd probably want pagination on that display there, except for the fact that there's a search box in there that allows searching by username, and we can't paginate with username search before these requests, because we only have user IDs to work with at that point....

Would probably need spectator server changes to at least send usernames across the wire. If not more.

[bdach]

Elsewhere:

Bancho allowed lookups this fast in order for this screen to exist. We probably need the osu-web end rate limit adjusted. Paginating would be a bit of a step backwards from people's expectations, I think.

@ppy/team-web would adjusting the rate limit be feasible?

For better context, this is the screen that is triggering this:

[nanaya]

the endpoint probably returns way too much data for what's being used there though. There's a /users/lookup-users but it's not available for api yet. I'd probably rather expose that one with higher limit instead.

[nanaya]

@bdach can you check if the data returned is enough? It's mainly missing statistics_rulesets compared to the /users one

example

```json { "users": [ { "avatar_url": "https://a.ppy.sh/2?1657169614.png", "country_code": "AU", "default_group": "default", "id": 2, "is_active": true, "is_bot": false, "is_deleted": false, "is_online": false, "is_supporter": true, "last_visit": "2024-09-25T07:18:12+00:00", "pm_friends_only": false, "profile_colour": "#3366FF", "username": "peppy", "country": { "code": "AU", "name": "Australia" }, "cover": { "custom_url": "https://assets.ppy.sh/user-profile-covers/2/baba245ef60834b769694178f8f6d4f6166c5188c740de084656ad2b80f1eea7.jpeg", "url": "https://assets.ppy.sh/user-profile-covers/2/baba245ef60834b769694178f8f6d4f6166c5188c740de084656ad2b80f1eea7.jpeg", "id": null }, "groups": [ { "colour": "#0066FF", "has_listing": false, "has_playmodes": false, "id": 33, "identifier": "ppy", "is_probationary": false, "name": "ppy", "short_name": "PPY", "playmodes": null }, { "colour": "#E45678", "has_listing": true, "has_playmodes": false, "id": 11, "identifier": "dev", "is_probationary": false, "name": "Developers", "short_name": "DEV", "playmodes": null } ] } ] } ```

[bdach]

As far as I can tell the client is using:

• ✔️ id
• ✔️ username
• ✔️ avatar_url
• ✔️ cover.url
• ✔️ country_code
• ✔️ last_visit
• ✔️ is_online
• ✔️ is_supporter
• ❌ support_level

groups is also good to have even if the client isn't using it (we really should be).

Looking at the web friends page which doesn't have support level shown, maybe we can do without support_level though...?

[nanaya]

I don't think support level is returned in the other endpoint either. Its value is calculated every time iirc.

[bdach]

Oh you're right actually. The client doesn't show any supporter markers on the currently online display, but it does on the friends display.

Yeah I think we can work with that set of fields then. We'll fix client to display supporter badge even if there is no level (is it even possible for API to return a user with {"is_supporter":1,"support_level":0}?)

[nanaya]

oh yeah the friend and group page do show it for "list mode" o_O

that's weird...

also no, the minimum value is 1 for current supporter.