search

aws-ia / terraform-aws-mendix-private-cloud

This AWS Partner Solution uses Terraform to deploy a Mendix infrastructure in the AWS Cloud.
Apache License 2.0
12 stars 14 forks source link

Failed module Prometheus #34

Closed eusse1102 closed 10 months ago

eusse1102 commented 1 year ago

Hi,

I am in the process of deployment and encountering an error with the Prometheus module. It's attempting to mount the prometheus-server onto a volume that doesn't exist, and I'm unsure why this is happening.

this error terraform:

│ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials │ │ with module.eks_blueprints_kubernetes_addons.module.prometheus[0].module.helm_addon.helm_release.addon[0], │ on .terraform\modules\eks_blueprints_kubernetes_addons\modules\kubernetes-addons\helm-addon\main.tf line 1, in resource "helm_release" "addon": │ 1: resource "helm_release" "addon" {

this error in k8s:

Warning ProvisioningFailed persistentvolumeclaim/prometheus-server failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-a10eae24-b883-4bf6-aa54-34a949921e1e": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-05a3611b1fd4caad9' does not exist....

Warning ProvisioningFailed persistentvolumeclaim/prometheus-server failed to provision volume with StorageClass "gp2": rpc error: code = AlreadyExists desc = Could not create volume "pvc-a10eae24-b883-4bf6-aa54-34a949921e1e": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)

danjhd commented 1 year ago

Is there any news on this issue? I encountered the same problem this week and it has blocked me as I cannot find any way to get past this problem. Is there a workarouns that anyone can suggest?

walkline commented 12 months ago

Can you please clarify which version you are using? I am unable to reproduce the issue with https://github.com/aws-ia/terraform-aws-mendix-private-cloud/releases/tag/v0.0.7.

danjhd commented 12 months ago

I was doing it by cloning the main branch of the repo, i assume that is the same as v0.0.7 but i can try again with that to be sure

danjhd commented 12 months ago

Tried again and same error. To be clear i am not certain about the K8s error as i am not sure how to look for that. But this is the error being shown from terraform command: │ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials │ │ with module.eks_blueprints_kubernetes_addons.module.ingress_nginx[0].module.helm_addon.helm_release.addon[0], │ on .terraform/modules/eks_blueprints_kubernetes_addons/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource "helm_release" "addon": │ 1: resource "helm_release" "addon" {

walkline commented 11 months ago

Unfortunately, I can't reproduce this issue locally or in CI. Have you tried cleaning up the environment and rerunning everything with a clean state?

danjhd commented 11 months ago

I am able to re-produce this with a BRAND NEW AWS account. I use a brand new EC2 instance to run the TF commands from so i know there is nothing "leftover" with TF either.... However, if i use an aWS account that is over 5 years old with a brand new EC2 instance for TF it works.... To me this seems to imply that the error message i am seeing is related to some AWS account setting that is not "set" in a new AWS account. However, i have very little TF or K8s knowledge so i am not able to get more information from the logs...