search

prometheus-community / helm-charts

Prometheus community Helm charts
Apache License 2.0
5.11k stars 5.02k forks source link

[kube-prometheus-stack] CRDs are not being installed by chart #1564

Closed ofiry-tm closed 2 years ago

ofiry-tm commented 2 years ago

Describe the bug a clear and concise description of what the bug is.

When I install the kube-prometheus-stack chart I get:

Error: UPGRADE FAILED: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

and if i manually deploy the CRDs that are in the prometheus-kube-stack (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/crds) :

kubectl -n myns apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl -n myns apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl -n myns apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl -n myns apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

then it seems to solve the issue.

but Isn't the kube-prometheus-stack chart should create these CRDs ?

how can I make the kube-prometheus-stack chart install the CRDs also ?

What's your helm version?

version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

21.0.4

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install prom prometheus-community/kube-prometheus-stack

Anything else we need to know?

No response

eugene-chernyshenko commented 2 years ago

pls reopen

ofiry-tm commented 2 years ago

pls reopen

done

or-navon commented 2 years ago

Applying the v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml has issue of too long annotation currently, executing: kubectl -n apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml kubectl -n replace -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml is a working workaround

monotek commented 2 years ago

Helm can only install CRDs once. See: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/

On updates you have to install them manually as described in the readme. See: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-33x-to-34x

mymasse commented 2 years ago

True about helm can only install CRDs but why not create a kubernetes job to update/install the CRDs, other charts do that with the help of:

annotations:
    "helm.sh/hook": post-install,post-upgrade,post-rollback
kfox1111 commented 2 years ago

True about helm can only install CRDs but why not create a kubernetes job to update/install the CRDs, other charts do that with the help of:

annotations:
    "helm.sh/hook": post-install,post-upgrade,post-rollback

If you do this, please please make it optional. Its a black box with a lot of permissions that's not easy to audit.

mymasse commented 2 years ago

The place where I got this from does make use of a flag to enable CRD upgrade so yeah should be something that is configurable.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

braunsonm commented 2 years ago

Bump. Not having this prevents truly automated updates.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

braunsonm commented 2 years ago

Still an issue

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

mymasse commented 2 years ago

Bump

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

sys-ops commented 2 years ago

Applying the v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml has issue of too long annotation currently, executing: kubectl -n apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml kubectl -n replace -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml is a working workaround

It looks like this workaround is still needed:

# kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
The CustomResourceDefinition "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

# kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.51.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created

# kubectl replace -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com replaced
hedeesaa commented 2 years ago

Still, we have problems on this side, Is there any specific reason behind this decision not to apply mandatory CRDs for kube-prometheus-stack in its helm chart?

Since there isn't any other charts for CRDs I believe we need to add them to the kube-prometheus-stack chart. Or create a new Chart for them.

monotek commented 2 years ago

They are part of the chart. See: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/crds

But helm can only install CRDs once and will not update them. See: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/

So update has to be done as described in the Readme: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-39x-to-40x

alifiroozi80 commented 1 year ago

In my case, I had to delete every CRD manually:

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd scrapeconfigs.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

Then install the Chart.

G-Novotny commented 9 months ago

Still an issue. Steps to reproduce:

  1. Create a kustomization.yaml:
cat <<'EOF' > kustomization.yaml
apiVersion:  kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: kustiomization-prometheus
namespace: monitoring

helmCharts:
- includeCrds: true
  name: kube-prometheus-stack
  releaseName: prometheus-stack
  repo: https://prometheus-community.github.io/helm-charts
EOF
  1. Apply it to the cluster:
kubectl kustomize  --enable-helm | kubectl apply -f -

Shell output:

customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
serviceaccount/prometheus-stack-grafana created
serviceaccount/prometheus-stack-grafana-test created
serviceaccount/prometheus-stack-kube-prom-admission created
serviceaccount/prometheus-stack-kube-prom-alertmanager created
serviceaccount/prometheus-stack-kube-prom-operator created
serviceaccount/prometheus-stack-kube-prom-prometheus created
serviceaccount/prometheus-stack-kube-state-metrics created
serviceaccount/prometheus-stack-prometheus-node-exporter created
role.rbac.authorization.k8s.io/prometheus-stack-grafana created
role.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrole created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrolebinding created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
configmap/prometheus-stack-grafana created
configmap/prometheus-stack-grafana-config-dashboards created
configmap/prometheus-stack-grafana-test created
configmap/prometheus-stack-kube-prom-alertmanager-overview created
configmap/prometheus-stack-kube-prom-apiserver created
configmap/prometheus-stack-kube-prom-cluster-total created
configmap/prometheus-stack-kube-prom-controller-manager created
configmap/prometheus-stack-kube-prom-etcd created
configmap/prometheus-stack-kube-prom-grafana-datasource created
configmap/prometheus-stack-kube-prom-grafana-overview created
configmap/prometheus-stack-kube-prom-k8s-coredns created
configmap/prometheus-stack-kube-prom-k8s-resources-cluster created
configmap/prometheus-stack-kube-prom-k8s-resources-multicluster created
configmap/prometheus-stack-kube-prom-k8s-resources-namespace created
configmap/prometheus-stack-kube-prom-k8s-resources-node created
configmap/prometheus-stack-kube-prom-k8s-resources-pod created
configmap/prometheus-stack-kube-prom-k8s-resources-workload created
configmap/prometheus-stack-kube-prom-k8s-resources-workloads-namespace created
configmap/prometheus-stack-kube-prom-kubelet created
configmap/prometheus-stack-kube-prom-namespace-by-pod created
configmap/prometheus-stack-kube-prom-namespace-by-workload created
configmap/prometheus-stack-kube-prom-node-cluster-rsrc-use created
configmap/prometheus-stack-kube-prom-node-rsrc-use created
configmap/prometheus-stack-kube-prom-nodes created
configmap/prometheus-stack-kube-prom-nodes-darwin created
configmap/prometheus-stack-kube-prom-persistentvolumesusage created
configmap/prometheus-stack-kube-prom-pod-total created
configmap/prometheus-stack-kube-prom-prometheus created
configmap/prometheus-stack-kube-prom-proxy created
configmap/prometheus-stack-kube-prom-scheduler created
configmap/prometheus-stack-kube-prom-workload-total created
secret/alertmanager-prometheus-stack-kube-prom-alertmanager created
secret/prometheus-stack-grafana created
secret/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-grafana created
service/prometheus-stack-kube-prom-alertmanager created
service/prometheus-stack-kube-prom-coredns created
service/prometheus-stack-kube-prom-kube-controller-manager created
service/prometheus-stack-kube-prom-kube-etcd created
service/prometheus-stack-kube-prom-kube-proxy created
service/prometheus-stack-kube-prom-kube-scheduler created
service/prometheus-stack-kube-prom-operator created
service/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-kube-state-metrics created
service/prometheus-stack-prometheus-node-exporter created
deployment.apps/prometheus-stack-grafana created
deployment.apps/prometheus-stack-kube-prom-operator created
deployment.apps/prometheus-stack-kube-state-metrics created
daemonset.apps/prometheus-stack-prometheus-node-exporter created
job.batch/prometheus-stack-kube-prom-admission-create created
job.batch/prometheus-stack-kube-prom-admission-patch created
alertmanager.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-config-reloaders created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-etcd created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-cpu-usage-second created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-cache created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-rss created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-swap created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-working-s created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-resource created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.pod-owner created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-availability.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-burnrate.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-histogram.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-slos created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-node-recording.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-state-metrics created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-apps created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-resources created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-storage created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-apiserver created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-controller-manager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kube-proxy created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kubelet created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-scheduler created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-network created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-grafana created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-apiserver created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-coredns created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-controller-manager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-etcd created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-proxy created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-state-metrics created
servicemonitor.monitoring.coreos.com/prometheus-stack-prometheus-node-exporter created
pod/prometheus-stack-grafana-test created
mutatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)

Here again the errors:

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)
jsantosa-minsait commented 9 months ago

Still an issue. Steps to reproduce:

1. Create a kustomization.yaml:
cat <<'EOF' > kustomization.yaml
apiVersion:  kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: kustiomization-prometheus
namespace: monitoring

helmCharts:
- includeCrds: true
  name: kube-prometheus-stack
  releaseName: prometheus-stack
  repo: https://prometheus-community.github.io/helm-charts
EOF
2. Apply it to the cluster:
kubectl kustomize  --enable-helm | kubectl apply -f -

Shell output:

customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
serviceaccount/prometheus-stack-grafana created
serviceaccount/prometheus-stack-grafana-test created
serviceaccount/prometheus-stack-kube-prom-admission created
serviceaccount/prometheus-stack-kube-prom-alertmanager created
serviceaccount/prometheus-stack-kube-prom-operator created
serviceaccount/prometheus-stack-kube-prom-prometheus created
serviceaccount/prometheus-stack-kube-state-metrics created
serviceaccount/prometheus-stack-prometheus-node-exporter created
role.rbac.authorization.k8s.io/prometheus-stack-grafana created
role.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrole created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrolebinding created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
configmap/prometheus-stack-grafana created
configmap/prometheus-stack-grafana-config-dashboards created
configmap/prometheus-stack-grafana-test created
configmap/prometheus-stack-kube-prom-alertmanager-overview created
configmap/prometheus-stack-kube-prom-apiserver created
configmap/prometheus-stack-kube-prom-cluster-total created
configmap/prometheus-stack-kube-prom-controller-manager created
configmap/prometheus-stack-kube-prom-etcd created
configmap/prometheus-stack-kube-prom-grafana-datasource created
configmap/prometheus-stack-kube-prom-grafana-overview created
configmap/prometheus-stack-kube-prom-k8s-coredns created
configmap/prometheus-stack-kube-prom-k8s-resources-cluster created
configmap/prometheus-stack-kube-prom-k8s-resources-multicluster created
configmap/prometheus-stack-kube-prom-k8s-resources-namespace created
configmap/prometheus-stack-kube-prom-k8s-resources-node created
configmap/prometheus-stack-kube-prom-k8s-resources-pod created
configmap/prometheus-stack-kube-prom-k8s-resources-workload created
configmap/prometheus-stack-kube-prom-k8s-resources-workloads-namespace created
configmap/prometheus-stack-kube-prom-kubelet created
configmap/prometheus-stack-kube-prom-namespace-by-pod created
configmap/prometheus-stack-kube-prom-namespace-by-workload created
configmap/prometheus-stack-kube-prom-node-cluster-rsrc-use created
configmap/prometheus-stack-kube-prom-node-rsrc-use created
configmap/prometheus-stack-kube-prom-nodes created
configmap/prometheus-stack-kube-prom-nodes-darwin created
configmap/prometheus-stack-kube-prom-persistentvolumesusage created
configmap/prometheus-stack-kube-prom-pod-total created
configmap/prometheus-stack-kube-prom-prometheus created
configmap/prometheus-stack-kube-prom-proxy created
configmap/prometheus-stack-kube-prom-scheduler created
configmap/prometheus-stack-kube-prom-workload-total created
secret/alertmanager-prometheus-stack-kube-prom-alertmanager created
secret/prometheus-stack-grafana created
secret/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-grafana created
service/prometheus-stack-kube-prom-alertmanager created
service/prometheus-stack-kube-prom-coredns created
service/prometheus-stack-kube-prom-kube-controller-manager created
service/prometheus-stack-kube-prom-kube-etcd created
service/prometheus-stack-kube-prom-kube-proxy created
service/prometheus-stack-kube-prom-kube-scheduler created
service/prometheus-stack-kube-prom-operator created
service/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-kube-state-metrics created
service/prometheus-stack-prometheus-node-exporter created
deployment.apps/prometheus-stack-grafana created
deployment.apps/prometheus-stack-kube-prom-operator created
deployment.apps/prometheus-stack-kube-state-metrics created
daemonset.apps/prometheus-stack-prometheus-node-exporter created
job.batch/prometheus-stack-kube-prom-admission-create created
job.batch/prometheus-stack-kube-prom-admission-patch created
alertmanager.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-config-reloaders created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-etcd created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-cpu-usage-second created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-cache created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-rss created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-swap created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-working-s created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-resource created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.pod-owner created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-availability.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-burnrate.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-histogram.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-slos created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-node-recording.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-state-metrics created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-apps created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-resources created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-storage created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-apiserver created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-controller-manager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kube-proxy created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kubelet created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-scheduler created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-network created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-grafana created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-apiserver created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-coredns created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-controller-manager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-etcd created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-proxy created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-state-metrics created
servicemonitor.monitoring.coreos.com/prometheus-stack-prometheus-node-exporter created
pod/prometheus-stack-grafana-test created
mutatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)

Here again the errors:

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)

I have the same issue too. Installing with helm directly and using the default values there is no error at all.

jsantosa-minsait commented 9 months ago

Still an issue. Steps to reproduce:

1. Create a kustomization.yaml:
cat <<'EOF' > kustomization.yaml
apiVersion:  kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: kustiomization-prometheus
namespace: monitoring

helmCharts:
- includeCrds: true
  name: kube-prometheus-stack
  releaseName: prometheus-stack
  repo: https://prometheus-community.github.io/helm-charts
EOF
2. Apply it to the cluster:
kubectl kustomize  --enable-helm | kubectl apply -f -

Shell output:

customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
serviceaccount/prometheus-stack-grafana created
serviceaccount/prometheus-stack-grafana-test created
serviceaccount/prometheus-stack-kube-prom-admission created
serviceaccount/prometheus-stack-kube-prom-alertmanager created
serviceaccount/prometheus-stack-kube-prom-operator created
serviceaccount/prometheus-stack-kube-prom-prometheus created
serviceaccount/prometheus-stack-kube-state-metrics created
serviceaccount/prometheus-stack-prometheus-node-exporter created
role.rbac.authorization.k8s.io/prometheus-stack-grafana created
role.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrole created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana created
rolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-grafana-clusterrolebinding created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-admission created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-prom-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-stack-kube-state-metrics created
configmap/prometheus-stack-grafana created
configmap/prometheus-stack-grafana-config-dashboards created
configmap/prometheus-stack-grafana-test created
configmap/prometheus-stack-kube-prom-alertmanager-overview created
configmap/prometheus-stack-kube-prom-apiserver created
configmap/prometheus-stack-kube-prom-cluster-total created
configmap/prometheus-stack-kube-prom-controller-manager created
configmap/prometheus-stack-kube-prom-etcd created
configmap/prometheus-stack-kube-prom-grafana-datasource created
configmap/prometheus-stack-kube-prom-grafana-overview created
configmap/prometheus-stack-kube-prom-k8s-coredns created
configmap/prometheus-stack-kube-prom-k8s-resources-cluster created
configmap/prometheus-stack-kube-prom-k8s-resources-multicluster created
configmap/prometheus-stack-kube-prom-k8s-resources-namespace created
configmap/prometheus-stack-kube-prom-k8s-resources-node created
configmap/prometheus-stack-kube-prom-k8s-resources-pod created
configmap/prometheus-stack-kube-prom-k8s-resources-workload created
configmap/prometheus-stack-kube-prom-k8s-resources-workloads-namespace created
configmap/prometheus-stack-kube-prom-kubelet created
configmap/prometheus-stack-kube-prom-namespace-by-pod created
configmap/prometheus-stack-kube-prom-namespace-by-workload created
configmap/prometheus-stack-kube-prom-node-cluster-rsrc-use created
configmap/prometheus-stack-kube-prom-node-rsrc-use created
configmap/prometheus-stack-kube-prom-nodes created
configmap/prometheus-stack-kube-prom-nodes-darwin created
configmap/prometheus-stack-kube-prom-persistentvolumesusage created
configmap/prometheus-stack-kube-prom-pod-total created
configmap/prometheus-stack-kube-prom-prometheus created
configmap/prometheus-stack-kube-prom-proxy created
configmap/prometheus-stack-kube-prom-scheduler created
configmap/prometheus-stack-kube-prom-workload-total created
secret/alertmanager-prometheus-stack-kube-prom-alertmanager created
secret/prometheus-stack-grafana created
secret/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-grafana created
service/prometheus-stack-kube-prom-alertmanager created
service/prometheus-stack-kube-prom-coredns created
service/prometheus-stack-kube-prom-kube-controller-manager created
service/prometheus-stack-kube-prom-kube-etcd created
service/prometheus-stack-kube-prom-kube-proxy created
service/prometheus-stack-kube-prom-kube-scheduler created
service/prometheus-stack-kube-prom-operator created
service/prometheus-stack-kube-prom-prometheus created
service/prometheus-stack-kube-state-metrics created
service/prometheus-stack-prometheus-node-exporter created
deployment.apps/prometheus-stack-grafana created
deployment.apps/prometheus-stack-kube-prom-operator created
deployment.apps/prometheus-stack-kube-state-metrics created
daemonset.apps/prometheus-stack-prometheus-node-exporter created
job.batch/prometheus-stack-kube-prom-admission-create created
job.batch/prometheus-stack-kube-prom-admission-patch created
alertmanager.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-config-reloaders created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-etcd created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-cpu-usage-second created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-cache created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-rss created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-swap created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-memory-working-s created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.container-resource created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-k8s.rules.pod-owner created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-availability.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-burnrate.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-histogram.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-apiserver-slos created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-general.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-prometheus-node-recording.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kube-state-metrics created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-apps created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-resources created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-storage created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-apiserver created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-controller-manager created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kube-proxy created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-kubelet created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-kubernetes-system-scheduler created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-exporter.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node-network created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-node.rules created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
prometheusrule.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-grafana created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-alertmanager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-apiserver created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-coredns created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-controller-manager created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-etcd created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-proxy created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kube-scheduler created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-kubelet created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-operator created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-prom-prometheus created
servicemonitor.monitoring.coreos.com/prometheus-stack-kube-state-metrics created
servicemonitor.monitoring.coreos.com/prometheus-stack-prometheus-node-exporter created
pod/prometheus-stack-grafana-test created
mutatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-stack-kube-prom-admission created
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)

Here again the errors:

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheusagents.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post prometheuses.monitoring.coreos.com)

Hi @G-Novotny use kubectl kustomize --enable-helm | kubectl create -f -

do not use apply, since it ne of the downsides of using kubectl apply is that it stores the entire spec as an annotation in the object (which it uses to understand how handle defaulted vs. deleted fields). What's happening is that data contained in fields is likely exceeding 262144 characters enforced by the K8s API server, and thus it cannot fit in the last-applied-configuration kubectl annotation.

Or check server-sideapply, https://foxutech.medium.com/how-to-fix-too-long-must-have-at-most-262144-bytes-in-argocd-2a00cddbbe99

Regards

iamraj007 commented 4 months ago

I too installed lasted prometheus community chart but it did not install any of the CRD's at all. Though i do see them code base of chart https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/charts/crds/crds But as we can see its seems the chart exceeding max characters enforced by the kubernetes API..

Any workaround or anyone got it working? 

 resource "helm_release" "prometheus-communityoperator" {
   name       = "prometheus-community"
   repository = "https://prometheus-community.github.io/helm-charts"
   chart      = "prometheus"
   version    = "25.22.0"
 }