Mangle Support for Amazon EKS

Anvesh42 commented 2 years ago

Mangle team, (@rpraveen-vmware @ashrimalivmware)

I communicated to you team earlier on https://github.com/vmware/mangle/issues/82 & https://github.com/vmware/mangle/issues/105 which were related to chaos setup on OpenShift cluster.

This time we are experiencing different issue. We are trying to have chaos functionality implemented in the AWS EKS environment for the GF applications and we are observing issues with kubeconfig integration. Here are the details,

Scenario

Mangle product is running on its own dedicated EKS cluster, lets says, EKS-Mangle
Target service (where chaos injection needs to happen) is running on another EKS cluster, let's say, EKS-Target
The kubeconfig file of EKS-Target,

EKS-Target-kubeconfig.txt

The required IAM policies and EKS RBAC policies have been established.

Problem Statement

While configuring & testing the endpoint connection on EKS-Mangle using the kubeconfig file of EKS-Target, we see failed connection error. Test Connection failed for endpoint, Please reverify the credentials {0} . Reason: Test Connection failed for endpoint, Please reverify the credentials

Our Initial RCA

We enabled the trace level logging on the mangle UI to grep the underlying root cause.
We found that the issue is related to AWS authentication from mangle. Mangle package doesn't have the AWS CLI or the AWS-IAM-Authenticator required to authenticate against AWS. Please see trace log below,

exec executable aws not found

mangle_eks_authentication_error

Some Useful Resources

From the available documentation on mangle, I do not see any specific use case support for EKS-mangle integration. Is this a limitation at this point?

Thanks Anvesh

rpraveen-vmware commented 2 years ago

Hi @Anvesh42 Thanks for reaching out to us. As we see that, to access the EKS Kubernetes cluster, the pre-requisite utilities are: kubectl, aws-iam-authenticator, awscli and priam.

-> Currently in our Mangle container, we have only the kubectl installed from the above list. Hence, we will not be able to access the resources of another EKS cluster from the present cluster.

-> We suggest you to deploy Mangle in the same cluster and access the resources locally (by adding K8s endpoint and namespace without kubeconfig).

-> However, we consider this as a feature request and try to address as part of our next Mangle release.

Let us know if any further queries. cc: @ashrimalivmware @george

Regards, Praveen R

Anvesh42 commented 2 years ago

@rpraveen-vmware Thanks for your response.

To your 2nd pointer, whether the endpoint is local or remote, the kubeconfig file field is imperative and must be defined.

In this case, the kube-system namespace is running within the same EKS cluster where mangle has been deployed and mangle expects the kubeconfig file. Please help understand when you say configure K8s endpoint without the kubeconfig file.

To your 3rd one, will you be able to provide us the timelines on the availability of mangle image that supports EKS workflows?

Thanks Anvesh

rpraveen-vmware commented 2 years ago

@Anvesh42 To answer the 2nd pointer: -> Go to "Add k8s Credential" and just give the "Name" of the k8s credential. Don't upload any kubeconfig file. In this case, use this k8sCredentialName while creating k8s Endpoint. Mangle considers it as local cluster when you don't provide kubeconfig file.

-> We discussed to have this as part of the next release of Mangle. Will update you on the timelines. cc: @ashrimalivmware

Anvesh42 commented 2 years ago

@rpraveen-vmware Thanks Praveen. That worked for us.

A quick question please - "Does cassandra DB configuration within the mangle product offer integration with AWS S3 or EBS? The default configuration is the NFS. Please see the image below. We wanted to see if cassandra storage can be moved to S3 or EBS instead of a filesystem.

Also, does mangle support other DB's such as mongodb & postgres? If so, will the team be able to provide us the supporting YAML's?

ashrimalivmware commented 2 years ago

Hi @Anvesh42, 1: Mangle doesn't offer any support with AWS S3 or EBS for Mangle DB (cassandra). Not sure if its possible to achieve using AWS capabilities. 2: Mangle DB only supports cassandra DB.

Thanks, -Avinash

Anvesh42 commented 2 years ago

@rpraveen-vmware @ashrimalivmware Due to current limitation concerning mangle-EKS integration, we are in the process of chaos testing design change. We have a question concerning the new design,

Does mangle support integration with cassandra given the scenario where mangle runs on one Kubernetes cluster while cassandra on another assuming firewall remains open between the two clusters?

Our current standard configuration:

Configuration we are trying to explore:

Please note the clusterB string in the cassandra contact point value.

We tried testing this approach and were able to find a hint of successful connection but couldn't be definite about it. We had an unsuccessful attempt trying to access mangle storage data on cassandra container (running on another cluster) via cqlsh command. Mangle documentation is limited in this area. Inputs are welcome.

Please let us know your thoughts on this design.

Thanks Anvesh

rpraveen-vmware commented 2 years ago

Hi @Anvesh42 This approach, to have DB in one cluster and the service in another cluster is something new that we have not tried off. We haven't tried out this deployment approach for our k8s.

-> Seeing that you had successful connection from Mangle to cassandra, need to check if it is really supported to provide the mount volume value of the DB in other cluster.

What's the reason behind this approach..? just curious to know.. cc: @ashrimalivmware

Anvesh42 commented 2 years ago

@rpraveen-vmware @ashrimalivmware

Do you think your team will be able to provide us the timelines at this point on EKS support for mangle product?

FYI, below are the 2 last comments from your team in this regard,

-> Currently in our Mangle container, we have only the kubectl installed from the above list. Hence, we will not be able to access the resources of another EKS cluster from the present cluster. -> However, we consider this as a feature request and try to address as part of our next Mangle release.

Thanks Anvesh

vmware / mangle