Masber commented 2 months ago

Instructions

At CSCS we heavily use BOS session templates to keep track clusters configurations, I was wondering if there is an interest from other centers in migrating BOS from CSM to OCHAMI in the same fashion we did with PCS?

Proposal Title

What needs to change:

Migrate a new microservice BOS from CSM into OCHAMI

What do you propose?

Migrate a new microservice BOS from CSM into OCHAMI

What alternatives exist?

BOS may be heavily integrated with K8s, if that is the case then, should an equivalent (more simple) microservice be created?

alexlovelltroy commented 1 month ago

Can you clarify how you use Session Templates and cluster configurations?

alexlovelltroy commented 1 month ago

@OpenCHAMI/tsc

Anyone have additional BOS Session Template discussion to add here? It would be good to see some examples and workflows surrounding them.

Masber commented 1 month ago

The idea is to create a single, centralized configuration template that holds both boot settings (BSS) and runtime settings (cloud-init?). These templates can be applied to HSM groups or partitions, replacing the need to manually modify BSS and cloud-init? for each machine or group.

Benefits:

Consistency: Templates ensure consistent configuration across machines.

Visibility: Easily check how machines are configured by viewing the template they’re using.

History Tracking: Templates can be versioned, allowing you to track configuration changes over time.

Efficiency: Reduces the overhead of manually updating BSS and cloud-init separately.

Then if you keep using those templates instead of modifying BSS and cloud-init? manually, you could easily check how machines are configured, also check the history in time

NOTE: I don't know how the OCHAMI cloud-init component works

alexlovelltroy commented 1 month ago

I'm assuming part of your interest here is in the features of manta that allow you to define a configuration on one cluster and then move that same configuration to a different cluster. I'm a big fan and definitely want all the same things you mention as benefits.

Having looked at the BOS service, I found it to be rather tightly coupled and couldn't figure out how to separate the configuration itself from the nodes it is/was applied to. It also appears to assume things about the deployment environment that aren't true in OpenCHAMI. Can you clarify which schemas in the API are most useful to you? I'm wondering if we can use feature flagging to incorporate a subset of BOS without adding all the dependencies of BOS.

Additionally, the OpenCHAMI cloud-init server has some features that the BSS cloud-init didn't have. We'll need to figure out if/how we want those also expressed in BOS (or any BOS alternative)

Masber commented 1 month ago

Yes, BOS is tightly coupled with CFS and the BOS concept could be redefined for OCHAMI.

The APIs I am interested in are:

sessiontemplate: to store all data related to boot and runtime configuration to be applied to a group of nodes
sessions: to trigger the configuration of a set of nodes based on a sessiontemplate
component: to keep track of current vs desired state of a cluster (eg if a sessiontemplate was applied)

Below I am providing an example of a schema: I know the schema below is very RDBM oriented and could be adapted.

eg of a schema:

struct template {
  id: string,
  created: datetime, // date and time when this template was created
  last_updated: datetime, // date and time when this template updated
  name: string,
  boot_configuration_id: string, // this would store BSS id
  runtime_configuration_id: string, // this would store the cloud_init id
}

struct session {
  id: string,
  created: datetime, // date and time when this template was created, this will tell sys admins when was the last time the cluster got configured
  template_id: string, // the template id this session is targeting
  group_name: string, // HSM group target for this session
  session_state: enum, // to keep track of the session
  error_message: string, // to store the stage in boot/configuration that failed and error message? maybe this field could be another struct?
  retry_count: int // number of retries in case the session failed previously
  completed_at: datetime
}

enum session_state {
  pending,
  running,
  completed,
  failed,
  cancelled
}

alexlovelltroy commented 1 month ago

BOS can be used without CFS, but BOS can't be used without Kubernetes which is a non-starter for OpenCHAMI.

The way I see it, we have several options:

Adopt a separate RFD to make Kubernetes a requirement for OpenCHAMI in order to support BOS
Modify BOS to work without Kubernetes
Create an API-compatible BOS service to handle templates and sessions the way CSM does

synackd commented 1 month ago

BOS can be used without CFS, but BOS can't be used without Kubernetes which is a non-starter for OpenCHAMI.

The way I see it, we have several options:
1. Adopt a separate RFD to make Kubernetes a requirement for OpenCHAMI in order to support BOS

2. Modify BOS to work without Kubernetes

3. Create an API-compatible BOS service to handle templates and sessions the way CSM does

Number 1 seems to go against the mission statement:

The mission of the OpenCHAMI consortium is to steward the collaborative development and continuous evolution of cloud-like software to manage High Performance Computing capacity regardless of the size or deployment platform.

(Emphasis mine)

If we are going to pursue BOS, either 2, 3, or an unlisted option seems like the way to go.

I haven't had experience using BOS, but @travisbcotton does; perhaps he could offer input as well.

alexlovelltroy commented 1 month ago

I agree that Option 1 is not compatible with our mission.

mharding-hpe commented 1 month ago

I've been asked to weigh in on the feasibility of option 2, and give an idea of how difficult it would be.

I'm going to limit my discussion to BOS v2, because BOS v1 is even more tightly coupled to Kubernetes.

There are two main technical challenges here:

BOS is written to run under Kubernetes
BOS is written assuming that it is on a CSM system, and thus assumes the presence of various other services (specifically: CFS, HSM, BSS, IMS, TAPMS, PCS, and S3)

Kubernetes

In the case of Kubernetes, the core BOS logic is essentially agnostic. Almost all of the BOS source code doesn't have any requirements about it. It does use Kubernetes charts and jobs during deployment and upgrades, and that would have to be addressed. Note that if the desire is not only to move away from Kubernetes but also move away from Docker, then it would be a bigger change. Again, the BOS code itself mostly would not care, but all of the code to build and deploy BOS assumes it's going to be running in a Docker container. There's no reason it HAS to be that way, though.

I will note that BOS relies on Kubernetes for both scaling and resiliency. If those things are important here, then they would need to be addressed on top of whatever other changes this would involve.

CSM service dependencies

BOS is constantly interacting with these other services, both the BOS API server and the BOS operators. The code is written in such a way that these service clients are mostly self-contained, but the parts of BOS which use them are everywhere. This is not to say one couldn't modify BOS to remove these things, or to modify their specifics.

Summary

On the one hand, there is no single big technical hurdle here (other than possibly the resiliency/scaling issue). Fundamentally, the core BOS logic could certainly work in a non-CSM, non-Kubernetes environment. But there would be a very large number of changes required, to handle all of the various differences. If the question is "Which is more work -- option 2 or option 3", then I think option 2 is more work. Or at best, a similar amount of work.

If it were me, I'd go for option 3. The only reason to lean toward option 2 in my view would be if it could be done in such a way that bug fixes and minor improvements made in CSM could make their way into the port without too much difficulty. Doing this would be possible, I think, but it may involve some code refactoring of BOS inside CSM, to enable a cleaner separation of the CSM-specific things.

jsl-hpe commented 1 month ago

As @mharding-hpe states, BOS sits at the top of most of the API stack and drives declarative state resulting in boot through lower level APIs (BSS, CFS, HMS Logical groups, power management (pcm), as well as hooks for things like artifacts stored in s3 and provisioning tech like iSCSI/DVS). Outside of k8-isms (hooks), any changes in the underlying APIs that BOS cares about will represent diff for the APIs that it talks to. Migrating BOSv2 to OpenChami will mean modifying all of this operator code to point at the correct/forked versions for state reconciliation.

The number of hooks that BOSv2 has within k8s is small, however the number of simultaneous operators running to provision nodes into a booted state are quite numerous.

We have found that the BOSv2 implementation to be satisfactory model for booting and provisioning nodes in CSM. It would be reasonable to expect that there is going to be a CRUD style API in future offerings with a very similar look and feel, with some notable differences that haven't been nailed down yet.

OpenCHAMI / roadmap

[RFD] Support BOS-style boot sessions in OpenCHAMI #73

Instructions

Proposal Title

What needs to change:

What do you propose?

What alternatives exist?

Kubernetes

CSM service dependencies

Summary