bmicklea commented 6 years ago

Goal

Enable 14k provisioned users on OpenShift.io. For active user concurrency requirements see https://github.com/openshiftio/openshift.io/issues/1146

Description

In order to handle ever-increasing numbers of provisioned users OpenShift.io will have to be able to idle and awake user pods to ensure that distribution of resources is optimized for active users. Below are two separate options that present an ideal (#1) and easier (#2) method.

Option 1 (Ideal): Aggressive Idling with Fast Awakening

Each pod (in stage or run, inclusive of Fabric8-generated PR pods) is idled after 15 minutes since last request. However, if a request is made to the pod's route then it is awakened and the request is connected.

[ ] Implement idling of stage and run pods at 15 minutes after last request.
[ ] Implement awakening of pods in response to an incoming request.
[ ] Notify the user that the pod is being rehydrated.
[ ] Add monitoring to warn of a situation in which a user is artificially keeping a pod active (e.g. by pinging it every 10 minutes).
[ ] PM: update terms of service that gives us the power to shut down / idle pods that we think are being misused (e.g. run pod that is being pinged every 10 minutes in order to keep it alive 24/7).

Option 2 (Easier): Static Idle Value

Idle pods that have not received a recent request in order to preserve capacity in the system for active users/accounts. Specific idle values are set via an API or PM-editable mechanism that doesn't require product re-release.

[ ] Implement idle trigger for stage, run and Fabric8 PR-generated pods.
[ ] Implement an API or other PM-controllable mechanism to set the idle separately for stage and run pods.
[ ] Notify the user that the pod is being rehydrated.
[ ] Add monitoring to warn of a situation in which a user is artificially keeping a pod active (e.g. by pinging it every 10 minutes).
[ ] PM: update terms of service that gives us the power to shut down / idle pods that we think are being misused (e.g. run pod that is being pinged every 10 minutes in order to keep it alive 24/7).

Parent Scenario

2028 OpenShift.io scales to meet user demand

joshuawilson commented 6 years ago

@kbsingh is this something you should own?

kbsingh commented 6 years ago

@joshuawilson happy to track, own as the deliverable.