Re-install Observatorium via ACM

[tumido]

When we first tried installing Observatorium via ACM we didn't read the docs properly and tried to deploy a Observatorium resource. Docs suggest that ACM has a wrapper on top of Observatorium required for this to work properly:

https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.2/html/observing_environments/observing-environments-intro#enable-observability

[tumido]

We want to deploy Observatorium on a managed cluster (not a management one, no MultiClusterHub deployed). We may need to follow the docs and do some changes to our Observatorium resource.

Initially we tried subscribing to the ACM operator on that cluster and deploy Observatorium directly. Naturally it didn't work because ACM probably requires MultiClusterObservability to be deployed instead (so the observatorium operator is deployed, etc).

[durandom]

cc @randymgeorge

@4n4nd can elaborate on the requirements that we expect and then we can dig into how much is met by ACM

But in short, we want to be able to store arbitrary metrics long term - not just those from OCP, but also from any workload. Basically Thanos as a service. Same for logs.

[tumido]

:heavy_plus_sign: we want to deploy that on a managed cluster (MOC Zero), not on the management cluster (MOC Infra).

[HumairAK]

Would this mean we need to deploy the MCO CR to have the observatorium operator available? Which, from what I gather from the docs above, will require us to configure some object store for it?

This may sound like a silly question but, at that point, can we not just deploy ACM on zero cluster, deploy MCO, and use that thanos?

[randymgeorge]

Agree on the statement of elaborate on the requirements. The discussion of what gets deployed where is confusing.

On Fri, Mar 12, 2021 at 9:14 AM Humair @.***> wrote:

Would this mean we need to deploy the MCO CR to have the observatorium operator available? Which, from what I gather from the docs above, will require us to configure some object store for it?

This may sound like a silly question but, at that point, can we not just deploy ACM on zero cluster, deploy MCO, and use that thanos?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/operate-first/apps/issues/365#issuecomment-797552992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7TKTCMDST4VMPZOP4AQ4TTDIOSTANCNFSM4ZCFLJ4Q .

--

Regards,

Randy George

Distinguished Engineer, Advanced Cluster Management

Red Hat https://www.redhat.com/

@.*** M: +15127511392 https://www.redhat.com/

[HumairAK]

Okay so to clarify, there are 2 clusters:

• moc-infra
• moc-zero

moc-infra is our management cluster, this is where we deploy acm/argocd. The acm in this cluster manages/deploys the other clusters (of which there is currently only the moc-zero cluster).

moc-zero is where we deploy our apps/services like odh. This is also where we currently have the observatorium operator deployed. We deploy an observatorium CR that provisions Thanos (see here). We were wondering if we can instead use acm for this?

[HumairAK]

The way I understand it we can do one of 2 things:

• Use ACM we already have on moc-infra cluster, enable observability by deploying an MCO and this will give us thanos, though it will be on the moc-infra cluster and not moc-zero, this can be problematic since this seems like adding too many heavy weight workloads on the infra cluster, and we want to keep things deployed to moc-infra minimal. However, it seems to me, by deploying to moc-infra we would be using the acm observability service to it's fullest, since this acm actually manages other c lusters.

• We can deploy another acm on the moc-zero cluster, but use it only to provision thanos, and not actually manage other clusters. Upside is that it keeps heavy workloads to the experimental cluster. Downside is that we don't get to benefit from pulling metrics from managed clusters.

Does that sound right?

[tumido]

Also MCO requires object storage being set up on the cluster. In case we want to deploy this to moc-infra we would need to either:

• Deploy local storage operator + OCS + we probably don't have that amount of storage available in there.
• Or connect it to storage cluster outside (IMHO better option)

[randymgeorge]

Per Marcel, what are the requirements. I am not sure why you are deploying Thanos on the managed cluster. Is this part of the application workload?

On Fri, Mar 12, 2021 at 10:54 AM Tomáš Coufal @.***> wrote:

Also MCO requires object storage being set up on the cluster. In case we want to deploy this to moc-infra we would need to either:

• Deploy local storage operator + OCS + we probably don't have that amount of storage available in there.
• Or connect it to storage cluster outside (IMGO better option I think)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/operate-first/apps/issues/365#issuecomment-797619076, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7TKTD5NHHNMHT6GRS6XMTTDJBNRANCNFSM4ZCFLJ4Q .

--

Regards,

Randy George

Distinguished Engineer, Advanced Cluster Management

Red Hat https://www.redhat.com/

@.*** M: +15127511392 https://www.redhat.com/

[randymgeorge]

It sounds like this is just part of the Open Data Hub workload and not related to operations. In that case, I would deploy it like any other application content. That can be via ACM or via ArgoCD. It was my misunderstanding. I thought that you were deploying Observatorium for the management which ACM already deploys an instance.

[tumido]

ACM marks the MultiClusterHub as a required resource. Therefore I'm a bit worried we can't setup "only" the Observatorium on a managed cluster, without setting it up as a management cluster as well.

[randymgeorge]

See the github where I commented. This is a use case where observatorium is part of the workload running on the cluster. ACM would not be the right approach.

On Mon, Mar 15, 2021 at 7:52 AM Tomáš Coufal @.***> wrote:

ACM marks the MultiClusterHub as a required resource. Therefore I'm a bit worried we can't setup "only" the Observatorium on a managed cluster, without setting it up as a management cluster as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/operate-first/apps/issues/365#issuecomment-799394228, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7TKTDK6RVXMERB6F3YD5DTDX7I3ANCNFSM4ZCFLJ4Q .

--

Regards,

Randy George

Distinguished Engineer, Advanced Cluster Management

Red Hat https://www.redhat.com/

@.*** M: +15127511392 https://www.redhat.com/

[4n4nd]

Current Monitoring Structure

• OperateFirst Clusters on MOC:

  • Infra
  • Zero

• MOC Infra Cluster:

  • Applications deployed:
  • ACM to set up User workload clusters. Example, Zero Cluster.
  • ArgoCD to manage User applications in User Workload Clusters.
  • Monitoring stack:
  • ArgoCd monitored/alerted using the Prometheus Stack in Zero Cluster
  • Openshift Monitoring and alerts for other Openshift cluster components.

• MOC Zero Cluster:

  • Applications deployed:
  • User Workload Applications like Jupyterhub, Data Catalog, Thoth apps etc.
  • Prometheus Operator to monitor Applications
  • Monitoring stack
  • Thanos using Observatorium Operator for long term storage of metric data.
  • Loki using Observatorium Operator for logs collection

[4n4nd]

The discussion was to see if we could replace Observatorium operator with ACM and use ACM Operator to deploy the Observatorium CR for MOC Zero Cluster monitoring.

[tumido]

Our target is to provide Observatorium as a service for users on the zero cluster only. Currently we don't have any intention to invest in Multi cluster observability managed from the moc-infra management cluster since it would require a lot of overhead for setting up things like storage on this (bare metal) cluster. The intention behind this issue was to look for an easier way to deploy Observatorium. This easier way appeared to be ACM however this turned out to be not a valid way forward since it requires making zero a management cluster as well. This is a technical obstacle.

If we want to move this forward we have 2 options:

• either change the architecture and redefine the Observatorium deliverable and deploy it on moc-infra cluster
• or collaborate with ACM on allowing Observatorium to be deployed as standalone via ACM operator only (No MultiClusterObservability or MultiClusterHub requirement).

Therefore I'm closing this issue for now, since either path requires discussion on greater scope - ACM is not offering us an "easier way" this time. Feel free to reopen if the status changes.