personalizable resource limits

[esraneufeld]

To avoid overscheduling of nodes, services should be scheduled with minimal resource requirements. In order not to limit "power users", those requirements can be dynamically be overwritten by users via the osparc user interface. Possibility to override is tied to users/group permissions. (computational backend for now, also keep in mind https://github.com/ITISFoundation/osparc-issues/issues/576)

### Tasks
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4515
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4516
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4357

### Sundae
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4468

[esraneufeld]

To avoid overscheduling of nodes, services should be scheduled with minimal resource requirements. In order not to limit "power users", those requirements can be dynamically be overwritten by users via the osparc user interface. Possibility to override is tied to users/group permissions. (computational backend for now, also keep in mind ITISFoundation/osparc-issues#576)

[sanderegg]

Update on sprint Macarons

Done

• Show services pre-defined resources to user (default resources):
  • https://github.com/ITISFoundation/osparc-simcore/pull/2987
  • https://github.com/ITISFoundation/osparc-simcore/pull/2998

Todo

• allow specific users to change limits on computational resources
  • save resources per node in project
  • handle sharing of projects
• if the resources are not available osparc should not run and show a clear message
• extend to dynamic services

[sanderegg]

Update on sprint Croissant

Done

• Manual assignment of computer nodes for specific users for dynamic-sidecar powered services: ITISFoundation/osparc-simcore#3034

Ongoing

Open

• Manual overriding services resources (reservations and limits)
• Allow specific users to change limits on computational services in osparc GUI
  • save resources per node in project
  • handle sharing of projects
• if resource not available osparc should not run and show a clear message
• extend to dynamic services

[elisabettai]

@mguidon, could something be done this sprint to prevent the case MS was hitting see this message in Mattermost? For the POs this is still very important and needed for s4l upcoming releases. Could you please discuss with the backend team and see what can be done?

[mguidon]

Hi. Strictly speaking, personalized resources are not required for s4l:web:lite. Melanies problem is related to her using a very outdated legacy service (with strict resource limits of outrageous 96 GB RAM) and her needing more RAM than the default one for the newer version (16GB). If we want to work on that I suggest to do a minimal thing that allows us to override the limits but not yet the users. (e.g. enhancing the services_specifications table in the db.) @sanderegg @GitHK any thoughts on this?

[sanderegg]

Goal for sprint Athena

• https://github.com/ITISFoundation/osparc-simcore/issues/3022

[sanderegg]

Update on sprint Athena

Done

• https://github.com/ITISFoundation/osparc-simcore/issues/3022

[sanderegg]

Goal for sprint Mithril

• [x] Allow service specifications (at least resources such as RAM, CPUs) override for computational services
• [x] deprecate MPI flag

[sanderegg]

Update on sprint Mithril

Done

• https://github.com/ITISFoundation/osparc-dask-gateway/pull/48
• https://github.com/ITISFoundation/osparc-dask-gateway/pull/49
• https://github.com/ITISFoundation/osparc-simcore/pull/3983
• https://github.com/ITISFoundation/osparc-simcore/pull/3989

[sanderegg]

Some notes

current implementation

• modern dynamic services and all computational services can have personalized resource limits/reservations
• these custom changes are set per user/group manually through osparc database
• these changes need access to the said database (dev-team or app-team) and to know what one is doing

If the user need to dynamically change a service resources I see the following requirements:

• UI to allow to change the resource requirements inside a study
• The allowed resource requirements depends on the user/group affiliations and access to actual computational resources (which might be known if running on internal cluster, or unknown if on an external cluster)
• the selected resource requirements shall be saved with the study for persistence
• One might have different default resources based on user/group affiliations
• These defaults will need to be set by some kind of administrator (the cluster's owner)
• When sharing a study with someone else, what happens when these resource requirements are not available to that user/group?

[sanderegg]

Goal for sprint Jelly Beans

Write down in details how this will be surfaced to the users

• Default resources
• Max resoruces
• How to link this to auto-scaling

Potential first use case: Melanie can choose how much RAM the ti jupyter smash has available. Default will be what is on the label. If the admin allows it, she can individually change the settings upon starting of the service.

[sanderegg]

Update on sprint Jelly Beans

Towards user-defined resources on services plan

1. Running a service with dynamically changed resources

• [x] a service maximum authorized resources per user/groups can be defined (database changes), if not defined would default to the current defaults (defined by service labels or overridden defaults)
• [x] a service defined resources are persisted with the project it is used in and is reloaded with the project
• [x] a service defined resources is visible in the UI
• [x] a service defined resources can be modified within a project up to the max authorized resources for the current user
• [x] a service defined resources is modifiable through the UI
• [x] the backend starts/runs the service based on service within project defined resources
• [ ] should be accessible from Public API as well

  2. administrator access to a service default resources

• [ ] administrator can override the default resources for a service (equivalent to current manual change in the database)
• [ ] administrator can set different default resources for different users/groups (equivalent to current manual change in the database
• [ ] administrator can change the maximum authorized used resources for a service for different users/groups
• [ ] system shall prevent administrator from completely using a shared machine by keeping some factor (e.g. if most powerful machine has 32 CPUs, then the maximum usable could be 8 CPUs)

  3. sharing a study

• [ ] pre-check if sharee has necessary available resources (similar as user rights)

Use-case:

• Melanie can choose how much RAM the ti jupyter smash has available. Default will be what is on the label. If the admin allows it, she can individually change the settings upon starting of the service.

after discussion with @mguidon the idea is to go with 1. first and wait with 2. until capabilities of dynamic clusters are better defined. 3. could be implemented once 1. is done.

[mguidon]

• Is the administrator one of us or a user? For the latter, how does a user become administrator?
• I think we should default to a selection of cpu/ram and gpu configs that can be chosen from (e.g. high cpu, high ram or similar)

[sanderegg]

• Should also be accessible from the PublicAPI

[sanderegg]

Update for sprint Pastel de Nata

Ongoing

• https://github.com/ITISFoundation/osparc-simcore/pull/4294

[sanderegg]

Update Watermelon

Done:

• https://github.com/ITISFoundation/osparc-simcore/pull/4417
• https://github.com/ITISFoundation/osparc-simcore/issues/4352
• https://github.com/ITISFoundation/osparc-simcore/issues/4354
• https://github.com/ITISFoundation/osparc-simcore/issues/4356
• https://github.com/ITISFoundation/osparc-simcore/pull/4335
• https://github.com/ITISFoundation/osparc-simcore/pull/4362
• https://github.com/ITISFoundation/osparc-simcore/pull/4384
• https://github.com/ITISFoundation/osparc-simcore/pull/4424
• https://github.com/ITISFoundation/osparc-simcore/pull/4428
• https://github.com/ITISFoundation/osparc-simcore/issues/4435
• https://github.com/ITISFoundation/osparc-simcore/issues/4436
• https://github.com/ITISFoundation/osparc-simcore/pull/4449

Summary: The user can now change a service required resources (CPUs, RAM) through the oSparc GUI. These changes are persisted within the project. There are currently no upper bound for resources. The platform will try to start the service using the defined required resources.

Open:

• PublicAPI changes to present the same options as the GUI
• Allow to select pre-defined EC2 instances to run services

[sanderegg]

Update Sundae

Ongoing

• https://github.com/ITISFoundation/osparc-simcore/issues/4468