Increase rate limit? (Attempt #2)

[ghost]

I figured this would be a good time to revisit #583. I've optimized everything I can on my end and I even took it a step further than that. I stopped using the stats endpoints for caching raid clears and PVP stats for my clan members because I have all their raw data in my database already. I'm computing the stats myself so the number of calls I'm making has been reduced for each user. I still use the stats endpoints to update the discord roles for non-clan member though. With these improvements, combined with the private public IP in the land of Azure, I don't currently have any problems.

I'm asking because within the next couple of weeks, I'm going to have my automated recruitment web page up. That will open up more traffic to my site. I want to have a dashboard showing the health of my clans (a resume of sorts) for people looking to join, but with that will come more API calls. These calls need to be server side because getting the quantity of pending users/invited users requires one of my admin accounts to get the information (to my knowledge). It'd be nice if GroupV2.GetMembersOfGroup returned the quantity of pending/invited users, but it doesn't so it takes an admin account to determine if a clan can accept new members.

The other factor is that with my recruitment automated, I'm going to be able to effortlessly expanding my clan past 7 divisions. Every time we expand, we fill up the clan quickly and we don't have problems keeping it full of people that meet our activity standards, so I honestly don't know how many clans we'll expand to. This is a factor because the more users I have in my clan, the more API calls I need to make to keep everyone's information up to date.

Is it possible to get a rate increase before I start having problems? My clan is as successful as it is because of the automation, so I'd prefer not having it impacted by rate limits.

[vthornheart-bng]

Unfortunately, I can't increase that rate limit - but I like some of the ideas you've mentioned about consolidating responses to reduce the amount of roundtrips you have to make. We've got pretty severe time constraints such that I don't know when we can get to them, but I think the idea is solid.

In the meantime - how frequently do you need to do that refresh? Perhaps you could consider doing the refresh of data less frequently?

[ghost]

If it comes down to it, I'll need to reduce the thread count on my jobs to slow things down. I'd prefer to speed things up though. It takes about 30 minutes for a new member to get their roles in my discord right now, so I'm going to need to write a method to load the data for just 1 person at a time to speed that up. When someone joins the clan with the automated recruitment I'll need to trigger that, but the role assigning code is pretty complicated and I think I'd rather increase the frequency of that job instead of trying to rework it to load data for just 1 person.

I suppose a concern of mine that is somewhat unrelated to this is if we expect the API to be down for days when black armory comes out. That'd make it so the most important time to have automated recruitment is also when the API is getting hit too hard to support it.

[vthornheart-bng]

I think for the latter concern, that's going to depend a lot on whether certain infrastructure improvements we've been making pay off. We will see about that - there's a lot of factors that go into API availability that are beyond my control, but we are working on it!

[ckhicks]

+1 to the idea for optimizing your automated recruitment functions to just process single users, if you can, @Akumati1435. From just reading through this with fresh eyes it sounds like that is a lot less processing as a whole to deal with just one single person, even if it is a bear to abstract in your app. Good stuff!

[ghost]

Well, the thing is, I'd still have the database updating every 30 minutes, but if the new people are loaded in a separate queue, I probably won't need to increase the frequency of the loaders.

Getting the character loader to load a single person won't be too bad. The clan history tracking is the only thing that might not work out of the box. The activity loader will work for a single person with 5 minutes of coding. The pvp stats part is already on a character by character basis, so I just need to move the part inside the for loop to another method and it's done. The raid data will require a new query, but that's about it. Linking the discord side to the bungie side.. I can just run that job as is whenever since it doesnt do much. That leaves the discord role assigning bit, which might be a slight problem because that code is terribly written. lol.