0hlov3 commented 2 years ago

We installed the Minio-Operator and the Minto-Tenant with the HELM-Charts, so we configured our chart to use the prometheusOperator, everything except the PrometheusScrapeConfig seems to work as expected in the Operator logs are messages like:

minio-operator-56c7757f45-r88xr operator E0817 11:19:26.040474       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:20:26.258768       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:21:26.474626       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:22:26.693781       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:23:26.909271       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:24:27.123383       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:25:27.337798       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:26:27.551345       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:27:27.764197       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:28:27.978694       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:29:28.195556       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:30:28.421171       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:31:28.639291       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:32:28.853166       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default
minio-operator-56c7757f45-r88xr operator E0817 11:33:29.067534       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default

Expected Behavior

As Described in crd.adoc prometheusOperator Directs the MinIO Operator to use prometheus operator. Tenant scrape configuration will be added to prometheus managed by the prometheus-operator.

As the Prometheus-Operator is located in the Monitoring Namespace it Seems that the Minio-Operator does not find the Prometheus-Operator.

Current Behavior

The Minio-Operator generates a Prometheus by itself and writes logs like:

minio-operator-56c7757f45-r88xr operator E0817 11:33:29.067534       1 main-controller.go:572] error syncing 'minio-tenant/minio-tenant': No prometheus found on namespace default

Possible Solution

Steps to Reproduce (for bugs)

Deploy the Tenant Helm-Chart with prometheusOperator: true

Context

Regression

Your Environment

Version used (minio-operator): v4.4.28
Environment name and version (e.g. kubernetes v1.17.2): Kubernetes v1.22.8
Server type and version:
Operating System and version (uname -a):

Link to your deployment file:


## Secret with default environment variable configurations to be used by MinIO Tenant.
## Not recommended for production deployments! Create the secret manually instead.
secrets:
name: minio1-env-configuration
# MinIO root user and password
accessKey: minio
secretKey: minio123

MinIO Tenant Definition

tenant:

Tenant name

name: minio1

Registry location and Tag to download MinIO Server image

image: repository: quay.io/minio/minio tag: RELEASE.2022-07-24T17-09-31Z pullPolicy: IfNotPresent

Customize any private registry image pull secret.

currently only one secret registry is supported

imagePullSecret: { }

If a scheduler is specified here, Tenant pods will be dispatched by specified scheduler.

If not specified, the Tenant pods will be dispatched by default scheduler.

scheduler: { }

Secret name that contains additional environment variable configurations.

The secret is expected to have a key named config.env containing environment variables exports.

configuration: name: minio1-env-configuration

Specification for MinIO Pool(s) in this Tenant.

pools:

Servers specifies the number of MinIO Tenant Pods / Servers in this pool.

## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
  ## custom name for the pool
  name: pool-0
  ## volumesPerServer specifies the number of volumes attached per MinIO Tenant Pod / Server.
  volumesPerServer: 2
  ## size specifies the capacity per volume
  size: 50Gi
  ## storageClass specifies the storage class name to be used for this pool
  storageClassName: standard
  ## Used to specify annotations for pods
  annotations: { }
  ## Used to specify labels for pods
  labels: { }
  ## Used to specify a toleration for a pod
  tolerations: [ ]
  ## nodeSelector parameters for MinIO Pods. It specifies a map of key-value pairs. For the pod to be
  ## eligible to run on a node, the node must have each of the
  ## indicated key-value pairs as labels.
  ## Read more here: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  nodeSelector: { }
  ## Affinity settings for MinIO pods. Read more about affinity
  ## here: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity.
  affinity: { }
  resources:
    requests:
      cpu: 2000m
      memory: 4096Mi 
    limits:
      cpu: 2000m
      memory: 4096Mi
  ## Configure security context
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    runAsNonRoot: false
    fsGroup: 0
  ## Configure topology constraints
  topologySpreadConstraints: [ ]
  ## Configure Runtime Class
  # runtimeClassName: ""

Mount path where PV will be mounted inside container(s).

mountPath: /export

Sub path inside Mount path where MinIO stores data.

subPath: /data

pool metrics to be read by Prometheus

metrics: enabled: false port: 9000 protocol: http certificate:

Use this field to provide one or more external CA certificates. This is used by MinIO

## to verify TLS connections with other applications:
## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
externalCaCertSecret: [ ]
## Use this field to provide a list of Secrets with external certificates. This can be used to configure
## TLS for MinIO Tenant pods. Create secrets as explained here:
## https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret
externalCertSecret:
  - name: tls-minio1-general
    type: kubernetes.io/tls
  - name: tls-minio1-star-general
    type: kubernetes.io/tls
  - name: tls-console-minio1-general
    type: kubernetes.io/tls
## Enable automatic Kubernetes based certificate generation and signing as explained in
## https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster
requestAutoCert: true
## This field is used only when "requestAutoCert" is set to true. Use this field to set CommonName
## for the auto-generated certificate. Internal DNS name for the pod will be used if CommonName is
## not provided. DNS name format is *.minio.default.svc.cluster.local
certConfig: { }

MinIO features to enable or disable in the MinIO Tenant

https://github.com/minio/operator/blob/master/docs/crd.adoc#features

features: bucketDNS: true domains: { }

List of bucket names to create during tenant provisioning

buckets: [ ]

List of secret names to use for generating MinIO users during tenant provisioning

users: [ ]

PodManagement policy for MinIO Tenant Pods. Can be "OrderedReady" or "Parallel"

Refer https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy

for details.

podManagementPolicy: Parallel

Liveness Probe for container liveness. Container will be restarted if the probe fails.

Refer https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.

liveness: { }

Readiness Probe for container readiness. Container will be removed from service endpoints if the probe fails.

Refer https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

readiness: { }

exposeServices defines the exposure of the MinIO object storage and Console services.

service is exposed as a loadbalancer in k8s service.

exposeServices: minio: true condole: true

kubernetes service account associated with a specific tenant

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

serviceAccountName: ""

Tenant scrape configuration will be added to prometheus managed by the prometheus-operator.

prometheusOperator: false

Enable JSON, Anonymous logging for MinIO tenants.

Refer https://github.com/minio/operator/blob/master/pkg/apis/minio.min.io/v2/types.go#L303

How logs will look:

$ k logs minio1-pool-0-0 -n default

{"level":"INFO","errKind":"","time":"2022-04-07T21:49:33.740058549Z","message":"All MinIO sub-systems initialized successfully"}

Notice they are in JSON format to be consumed

logging: anonymous: true json: true quiet: true

serviceMetadata allows passing additional labels and annotations to MinIO and Console specific

services created by the operator.

serviceMetadata: { }

Add environment variables to be set in MinIO container (https://github.com/minio/minio/tree/master/docs/config)

env:

name: MINIO_DOMAIN value: "minio.REDACTED"
name: MINIO_BROWSER_REDIRECT_URL value: "https://console-minio.REDACTED"
name: MINIO_SERVER_URL value: "https://minio.REDACTED"
name: MINIO_API_SECURE_CIPHERS value: "off"
PriorityClassName indicates the Pod priority and hence importance of a Pod relative to other Pods.

This is applied to MinIO pods only.

Refer Kubernetes documentation for details https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass/

priorityClassName: ""

Define configuration for KES (stateless and distributed key-management system)

Refer https://github.com/minio/kes

kes:

image: "" # minio/kes:v0.18.0

env: [ ]

replicas: 2

configuration: |-

address: :7373

root: _ # Effectively disabled since no root identity necessary.

tls:

key: /tmp/kes/server.key # Path to the TLS private key

cert: /tmp/kes/server.crt # Path to the TLS certificate

proxy:

identities: []

header:

cert: X-Tls-Client-Cert

policy:

my-policy:

paths:

- /v1/key/create/*

- /v1/key/generate/*

- /v1/key/decrypt/*

identities:

- ${MINIO_KES_IDENTITY}

cache:

expiry:

any: 5m0s

unused: 20s

log:

error: on

audit: off

keys:

KES configured with fs (File System mode) doesnt work in Kubernetes environments and it's not recommended

use a real KMS

fs:

path: "./keys" # Path to directory. Keys will be stored as files. Not Recommended for Production.

vault:

endpoint: "http://vault.default.svc.cluster.local:8200" # The Vault endpoint

namespace: "" # An optional Vault namespace. See: https://www.vaultproject.io/docs/enterprise/namespaces/index.html

prefix: "my-minio" # An optional K/V prefix. The server will store keys under this prefix.

approle: # AppRole credentials. See: https://www.vaultproject.io/docs/auth/approle.html

id: "" # Your AppRole Role ID

secret: "" # Your AppRole Secret ID

retry: 15s # Duration until the server tries to re-authenticate after connection loss.

tls: # The Vault client TLS configuration for mTLS authentication and certificate verification

key: "" # Path to the TLS client private key for mTLS authentication to Vault

cert: "" # Path to the TLS client certificate for mTLS authentication to Vault

ca: "" # Path to one or multiple PEM root CA certificates

status: # Vault status configuration. The server will periodically reach out to Vault to check its status.

ping: 10s # Duration until the server checks Vault's status again.

aws:

The AWS SecretsManager key store. The server will store

secret keys at the AWS SecretsManager encrypted with

AWS-KMS. See: https://aws.amazon.com/secrets-manager

secretsmanager:

endpoint: "" # The AWS SecretsManager endpoint - e.g.: secretsmanager.us-east-2.amazonaws.com

region: "" # The AWS region of the SecretsManager - e.g.: us-east-2

kmskey: "" # The AWS-KMS key ID used to en/decrypt secrets at the SecretsManager. By default (if not set) the default AWS-KMS key will be used.

credentials: # The AWS credentials for accessing secrets at the AWS SecretsManager.

accesskey: "" # Your AWS Access Key

secretkey: "" # Your AWS Secret Key

token: "" # Your AWS session token (usually optional)

imagePullPolicy: "IfNotPresent"

externalCertSecret: null

clientCertSecret: null

Key name to be created on the KMS, default is "my-minio-key"

keyName: ""

resources: { }

nodeSelector: { }

affinity:

nodeAffinity: { }

podAffinity: { }

podAntiAffinity: { }

tolerations: [ ]

annotations: { }

labels: { }

serviceAccountName: ""

securityContext:

runAsUser: 1000

runAsGroup: 1000

runAsNonRoot: true

fsGroup: 1000

Prometheus setup for MinIO Tenant.

prometheus: image: "" # defaults to quay.io/prometheus/prometheus:latest env: [ ] sidecarimage: "" # defaults to alpine initimage: "" # defaults to busybox:1.33.1 diskCapacityGB: 1 storageClassName: standard annotations: { } labels: { } nodeSelector: { } affinity: nodeAffinity: { } podAffinity: { } podAntiAffinity: { } resources: { } serviceAccountName: "" securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000

LogSearch API setup for MinIO Tenant.

log: image: "" # defaults to minio/operator:v4.4.17 env: [ ] resources: { } nodeSelector: { } affinity: nodeAffinity: { } podAffinity: { } podAntiAffinity: { } tolerations: [ ] annotations: { } labels: { } audit: diskCapacityGB: 1

Postgres setup for LogSearch API

db: image: "" # defaults to library/postgres env: [ ] initimage: "" # defaults to busybox:1.33.1 volumeClaimTemplate: metadata: { } spec: storageClassName: standard accessModes:
- ReadWriteOnce resources: requests: storage: 1Gi resources: { } nodeSelector: { } affinity: nodeAffinity: { } podAffinity: { } podAntiAffinity: { } tolerations: [ ] annotations: { } labels: { } serviceAccountName: "" securityContext: runAsUser: 999 runAsGroup: 999 runAsNonRoot: true fsGroup: 999 serviceAccountName: "" securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000

ingress: api: enabled: false ingressClassName: "" labels: { } annotations: { } tls: [ ] host: minio.local path: / pathType: Prefix console: enabled: false ingressClassName: "" labels: { } annotations: { } tls: [ ] host: minio-console.local path: / pathType: Prefix

cyril-corbon commented 2 years ago

You just have to set PROMETHEUS_NAMESPACE as environment variable to the minio-controller deployments. sample:

  env:
    - name: PROMETHEUS_NAMESPACE
      value: "monitoring"

it's documented there : https://github.com/minio/operator/blob/6cf1612e9b64a6b400394ac5f01353953d58fa37/UPGRADE.md#v439---v440

0hlov3 commented 2 years ago

Thank you very much, I must have searched for 4 hours and just couldn't find it or overlooked it.

gecube commented 2 years ago

@cyril-corbon Hi! Thanks for your advice. It is very important and useful. But I'd like to see the explicit option to select the Prometheus namespace in the HELM chart of minio-operator. Because it is really ambiguos to find the solution. Thanks Gods for this nice issue tracker!!!

everythings-gonna-be-alright commented 1 month ago

error syncing 'minio-tenant/minio': No prometheus found on namespace victoria-metrics

everythings-gonna-be-alright commented 1 month ago

Users can build a monitoring system not only based on the Prometheus operator. They can also use VictoriaMetrics or the Grafana stack. Anyway, it looks like it's not a big deal. It is possible to generate config manually. Fortunately, the operator source code looks nice and easy to understand, so I found out how it generates that configuration in ~5 minutes. Hotfix will look like this:

      - job_name: 'minio'
        scheme: https
        metrics_path: "/minio/v2/metrics/cluster"
        bearer_token: "your_token"
        tls_config:
          insecure_skip_verify: true
        static_configs:
          - targets: ["minio-hl.minio-tenant.svc.cluster.local:9000"]

bearer token you can create like in this piece of code: https://github.com/minio/operator/blob/master/pkg/apis/minio.min.io/v2/helper.go#L398

minio / operator

Operator Logs: No prometheus found on namespace default #1244

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment

MinIO Tenant Definition

Tenant name

Registry location and Tag to download MinIO Server image

Customize any private registry image pull secret.

currently only one secret registry is supported

If a scheduler is specified here, Tenant pods will be dispatched by specified scheduler.

If not specified, the Tenant pods will be dispatched by default scheduler.

Secret name that contains additional environment variable configurations.

The secret is expected to have a key named config.env containing environment variables exports.

Specification for MinIO Pool(s) in this Tenant.

Servers specifies the number of MinIO Tenant Pods / Servers in this pool.

Mount path where PV will be mounted inside container(s).

Sub path inside Mount path where MinIO stores data.

pool metrics to be read by Prometheus

Use this field to provide one or more external CA certificates. This is used by MinIO

MinIO features to enable or disable in the MinIO Tenant

https://github.com/minio/operator/blob/master/docs/crd.adoc#features

List of bucket names to create during tenant provisioning

List of secret names to use for generating MinIO users during tenant provisioning

PodManagement policy for MinIO Tenant Pods. Can be "OrderedReady" or "Parallel"

Refer https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy

for details.

Liveness Probe for container liveness. Container will be restarted if the probe fails.

Refer https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.

Readiness Probe for container readiness. Container will be removed from service endpoints if the probe fails.

Refer https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

exposeServices defines the exposure of the MinIO object storage and Console services.

service is exposed as a loadbalancer in k8s service.

kubernetes service account associated with a specific tenant

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Tenant scrape configuration will be added to prometheus managed by the prometheus-operator.

Enable JSON, Anonymous logging for MinIO tenants.

Refer https://github.com/minio/operator/blob/master/pkg/apis/minio.min.io/v2/types.go#L303

How logs will look:

$ k logs minio1-pool-0-0 -n default

{"level":"INFO","errKind":"","time":"2022-04-07T21:49:33.740058549Z","message":"All MinIO sub-systems initialized successfully"}

Notice they are in JSON format to be consumed

serviceMetadata allows passing additional labels and annotations to MinIO and Console specific

services created by the operator.

Add environment variables to be set in MinIO container (https://github.com/minio/minio/tree/master/docs/config)

PriorityClassName indicates the Pod priority and hence importance of a Pod relative to other Pods.

This is applied to MinIO pods only.

Refer Kubernetes documentation for details https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass/

Define configuration for KES (stateless and distributed key-management system)

Refer https://github.com/minio/kes

kes:

image: "" # minio/kes:v0.18.0

env: [ ]

replicas: 2

configuration: |-

address: :7373

root: _ # Effectively disabled since no root identity necessary.

tls:

key: /tmp/kes/server.key # Path to the TLS private key

cert: /tmp/kes/server.crt # Path to the TLS certificate

proxy:

identities: []

header:

cert: X-Tls-Client-Cert

policy:

my-policy:

paths:

- /v1/key/create/*

- /v1/key/generate/*

- /v1/key/decrypt/*

identities:

- ${MINIO_KES_IDENTITY}

cache:

expiry:

any: 5m0s

unused: 20s