What does this pull request do? Explain your changes. (required)
Draft of ai-video selection algo fix.
Suspension was not working because the penalty was always 3. This logic was a carryover from transcoding where the suspender always started at a refresh count of 0 because a new session manager was created with each stream. For AI, we are reusing the session manager and the suspender so the refresh count does not reset between requests. The fix to suspension is to consider the current refresh count when calculating the penalty so it is 3 more than the current refresh count in the suspender.
There was also an issue where the discoveryPoolSize was always 100 and with limited orchestrators providing models a refresh of sessions was being done with every request. I added an initialPoolSize field to track the last refresh pool size to use with the shouldRefreshSessions logic rather than 100. This stabilizes the suspender to allow more orchestrators to be tried with each Select call.
Last update was moving the signalRefresh() for the suspender that increments the refresh counter in the suspender to the Refresh function makes it more stable that every time we refresh sessions we add to the suspender refresh count
Happy to segregate some of these changes to separate PRs. The suspension fixes can be added separately without dependency on ai-worker PR.
Specific updates (required)
Updates suspender to use the current refresh count of the suspender in the selector.
Moves penalty to the AISessionSelector to make it easier to update and available for calculations on the suspension needed
releases all Os when there are none in the warm and cold pool
Adds option to not use managed containers.
How did you test each of these updates (required)
I have been running these updates on my gateway. Tested 1-200 requests with 5-10 workers sending to gateway. All completed with 1-2 orchestrators providing Bytedance model.
What does this pull request do? Explain your changes. (required) Draft of ai-video selection algo fix.
Suspension was not working because the
penaltywas always3. This logic was a carryover from transcoding where the suspender always started at a refresh count of 0 because a new session manager was created with each stream. For AI, we are reusing the session manager and the suspender so the refresh count does not reset between requests. The fix to suspension is to consider the current refresh count when calculating the penalty so it is 3 more than the current refresh count in the suspender.There was also an issue where the
discoveryPoolSizewas always 100 and with limited orchestrators providing models a refresh of sessions was being done with every request. I added aninitialPoolSizefield to track the last refresh pool size to use with theshouldRefreshSessionslogic rather than 100. This stabilizes the suspender to allow more orchestrators to be tried with eachSelectcall.Last update was moving the
signalRefresh()for the suspender that increments the refresh counter in the suspender to theRefreshfunction makes it more stable that every time we refresh sessions we add to the suspender refresh countHappy to segregate some of these changes to separate PRs. The suspension fixes can be added separately without dependency on ai-worker PR.
Specific updates (required)
How did you test each of these updates (required)
I have been running these updates on my gateway. Tested 1-200 requests with 5-10 workers sending to gateway. All completed with 1-2 orchestrators providing Bytedance model.
Does this pull request close any open issues?
Checklist:
makeruns successfully./test.shpass