Need help with k8s

[hyfj44255]

Could you help me implement a k8s operator called "external secrets" using kubebuilder? Here's the detail of it: This k8s operator have two APIs:"secretstores" and "externalsecrets". Secretstores is used for talk to different kind of endpoints to fetch secret information. We consider endpoint as "provider",the endpoints could be, for instance - Azure Key Vault . For now there are two kinds of providers, "Valut" and "Mercury" ,for valut we can just leave it as skull, we just need to implement the other provider - "Mercury" which is a token management service. To access it,first we need to specify auth token endpoint and the k8s secret going to be referenced as user and password to get bearer token, then we can use this bearer token to fetch secret from a secret entity endpoint we specified. So in provider Mercury we need to specify the secret that contains user and password and the endpoint for get bearer token and the endpoint for fetch secret entities. Another API in this CRD is the externalsecrets, in this CRD we need to specify which Secretstores it references to ,the app name and environment as parameter which Secretstores need for fetch token from endpoint. In reconcile loop the externalsecrets' controller will create a k8s secrets which contains the content get from Mercury for us. We can set the refresh interval in externalsecrets, the secret will be synced accordingly.

[alabulei1]

Sure, I can help you with that! Here are the steps to implement the "external secrets" operator using kubebuilder:

1. Install kubebuilder and set it up.
2. Use kubebuilder to create a new api called "secretstore".
3. Define the spec and status for "secretstore". The spec will include the provider (Mercury), the k8s secret for user and password, the auth token endpoint and the secret entity endpoint.
4. Use kubebuilder to create a new api called "externalsecrets". This api will reference the "secretstore" api.
5. Define the spec and status for "externalsecrets". The spec will include the app name, environment, and the secretstore reference. Add a refresh interval for syncing the secret.
6. Run the make command to generate the code.
7. Implement the reconciler for "secretstore" and "externalsecrets".
8. Test the operator on your kubernetes cluster.

I hope this helps you get started with implementing the "external secrets" operator using kubebuilder. Let me know if you need any further assistance!

[hyfj44255]

Could you provide more detail ?

[alabulei1]

Sure, I'd be happy to provide more detail on each step. Here is a more in-depth guide on implementing the "external secrets" operator using kubebuilder:

Step 1: Install kubebuilder and set it up.

Kubebuilder is a toolkit for building Kubernetes APIs using best practices. Here is how to install and set it up:

1. Download Kubebuilder from their official website https://book.kubebuilder.io/quick-start.html#installation.
2. Install Kubebuilder by unzipping the archive and adding the bin folder to your system path.
3. Run the "create api" command to scaffold your API.

Step 2: Use kubebuilder to create a new api called "secretstore".

1. Open a terminal and navigate to the directory where you want to create the new operator.
2. Run the following command to create the api:

kubebuilder create api --group secrets --version v1alpha1 --kind SecretStore

This command will create a new API called "SecretStore" in the pkg/apis/secrets/v1alpha1 directory.

Step 3: Define the spec and status for "secretstore".

1. Open the pkg/apis/secrets/v1alpha1/secretstore_types.go file and define the spec and status for your API.

type SecretStoreSpec struct {
    Provider      string               `json:"provider"`
    K8sSecretName string               `json:"k8sSecretName"`
    AuthTokenURL  string               `json:"authTokenUrl"`
    EntityURL     string               `json:"entityUrl"`
}

type SecretStoreStatus struct {
    SecretName string               `json:"secretName"`
    Conditions []metav1.Condition   `json:"conditions"`
}

2. Add validation tags for each field in the spec struct to ensure that the required fields are provided.

kubebuilder provides a go:generate tag in the source code that you can run to generate code for your API based on your spec.

To generate the struct validation code from the tags, run the following command:

go generate ./...

Step 4: Use kubebuilder to create a new api called "externalsecrets".

1. Open a terminal and navigate to the directory where you want to create the new operator.
2. Run the following command to create the api:

kubebuilder create api --group secrets --version v1alpha1 --kind ExternalSecret

This command will create a new API called "ExternalSecret" in the pkg/apis/secrets/v1alpha1 directory.

Step 5: Define the spec and status for "externalsecrets".

1. Open the pkg/apis/secrets/v1alpha1/externalsecret_types.go file and define the spec and status for your API.

type ExternalSecretSpec struct {
    AppName            string   `json:"appName"`
    Environment        string   `json:"environment"`
    SecretStoreRef     string   `json:"secretStoreRef"`
    RefreshIntervalSec int      `json:"refreshIntervalSec"`
}

type ExternalSecretStatus struct {
    SecretName  string             `json:"secretName"`
    Conditions []metav1.Condition `json:"conditions"`
}

2. Add validation tags for each field in the spec struct to ensure that the required fields are provided.

Step 6: Run the make command to generate the code.

1. Run the following command to generate the controller and webhook code:

make

2. This command will generate the necessary code for building the operator.

Step 7: Implement the reconciler for "secretstore" and "externalsecrets".

1. Open the pkg/controller/secretstore/secretstore_controller.go file and implement the reconciler for secretstore.

Here's an example of what the reconciler code could look like:

func (r *SecretStoreReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("secretstore", req.NamespacedName)

    // your reconciler logic here

    return ctrl.Result{}, nil
}

2. Open the pkg/controller/externalsecret/externalsecret_controller.go file and implement the reconciler for externalsecret.

Here's an example of what the reconciler code could look like:

func (r *ExternalSecretReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("externalsecret", req.NamespacedName)

    // your reconciler logic here

    return ctrl.Result{}, nil
}

Step 8: Test the operator on your kubernetes cluster.

1. Build the operator with the following command:

make docker-build IMG=<your-docker-registry>/external-secrets-operator:v0.0.1

2. Push the operator image to your docker registry:

docker push <your-docker-registry>/external-secrets-operator:v0.0.1

3. Deploy your operator to your Kubernetes cluster:

make deploy IMG=<your-docker-registry>/external-secrets-operator:v0.0.1

4. Verify that the operator is running correctly by checking the logs:

kubectl logs -f deployment/external-secrets-operator-manager -n external-secrets-operator-system

5. Create an instance of the ExternalSecret API to test the operator:

apiVersion: secrets.example.com/v1alpha1
kind: ExternalSecret
metadata:
  name: example-externalsecret
spec:
  appName: my-app
  environment: staging
  secretStoreRef: my-secret-store
  refreshIntervalSec: 3600

6. Verify that a new secret has been created for your ExternalSecret instance:

kubectl get secret

I hope this guide helps you implement the "external secrets" operator using kubebuilder. Good luck! Let me know if you have any further questions.

[hyfj44255]

could you implement the SecretStoreReconciler and ExternalSecretReconciler for me?

[alabulei1]

Sure, I'd be happy to provide a rough implementation example for both the SecretStoreReconciler and ExternalSecretReconciler. Here's an example implementation for each:

SecretStoreReconciler

type SecretStoreReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

func (r *SecretStoreReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("secretstore", req.NamespacedName)

    var ss secretsv1alpha1.SecretStore
    if err := r.Get(ctx, req.NamespacedName, &ss); err != nil {
        log.Error(err, "unable to fetch SecretStore")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    var s corev1.Secret
    if err := r.Get(ctx, types.NamespacedName{Name: ss.Spec.K8sSecretName, Namespace: req.Namespace}, &s); err != nil {
        log.Error(err, "unable to fetch secret")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Your external secret provider logic here

    return ctrl.Result{}, nil
}

func (r *SecretStoreReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&secretsv1alpha1.SecretStore{}).
        Complete(r)
}

ExternalSecretReconciler

type ExternalSecretReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

func (r *ExternalSecretReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("externalsecret", req.NamespacedName)

    var es secretsv1alpha1.ExternalSecret
    if err := r.Get(ctx, req.NamespacedName, &es); err != nil {
        log.Error(err, "unable to fetch ExternalSecret")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    var ss secretsv1alpha1.SecretStore
    if err := r.Get(ctx, types.NamespacedName{Name: es.Spec.SecretStoreRef, Namespace: req.Namespace}, &ss); err != nil {
        log.Error(err, "unable to fetch SecretStore")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Your external secret provider logic here
    // Use the data from the SecretStore and ExternalSecret
    // to create or update the corresponding Kubernetes secret

    return ctrl.Result{}, nil
}

func (r *ExternalSecretReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&secretsv1alpha1.ExternalSecret{}).
        Complete(r)
}

Note that these implementations are just examples of what the reconcilers implementation could look like. You'll need to replace the comments with your actual implementation logic based on your provider and use case. Additionally, you may need to add additional error handling and validation code as needed.

I hope this helps! Let me know if you have any further questions.

[hyfj44255]

I think you can do more ,please take https://github.com/external-secrets/external-secrets for reference

[alabulei1]

Yes, definitely! If you would like a reference implementation of the SecretStoreReconciler and ExternalSecretReconciler using the external-secrets project, here's an example:

import (
    "context"
    "encoding/base64"
    "fmt"
    "strings"

    "github.com/go-logr/logr"
    "github.com/google/go-cmp/cmp"
    "github.com/pkg/errors"
    "github.com/prometheus/common/log"
    "github.com/redhat-cop/external-secrets/controllers/providers"
    "github.com/redhat-cop/external-secrets/e2e/framework"
    "github.com/redhat-cop/external-secrets/e2e/utils"
    "github.com/redhat-cop/external-secrets/pkg/apis"
    externalv1alpha1 "github.com/redhat-cop/external-secrets/pkg/apis/externalsecrets/v1alpha1"
    "github.com/redhat-cop/external-secrets/pkg/provider"
    "github.com/redhat-cop/operator-utils/pkg/util"
    "k8s.io/apimachinery/pkg/api/errors"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/handler"
    "sigs.k8s.io/controller-runtime/pkg/manager"
    "sigs.k8s.io/controller-runtime/pkg/predicate"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
    "sigs.k8s.io/controller-runtime/pkg/source"
)

// SecretStoreReconciler reconciles SecretStore objects
type SecretStoreReconciler struct {
    Client client.Client
    Log    logr.Logger
    // Add secret provider here
}

func (r *SecretStoreReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("SecretStore", req.NamespacedName)
    log.Info(fmt.Sprintf("Reconciling Secret Store : %s/%s", req.NamespacedName.Namespace, req.NamespacedName.Name))

    // Fetch the SecretStore instance
    secretStore := &apis.SecretStore{}
    if err := r.Client.Get(ctx, req.NamespacedName, secretStore); err != nil {
        if errors.IsNotFound(err) {
            provider.UnregisterStore(secretStore)
            return ctrl.Result{}, nil
        }
        log.Error(err, "unable to fetch SecretStore")
        return ctrl.Result{}, err
    }

    if secretStore.ObjectMeta.DeletionTimestamp.IsZero() {
        // register secret store
        err := provider.RegisterStore(*secretStore, r.Provider())
        if err != nil {
            log.Error(err, "unable to register SecretStore")
            return ctrl.Result{}, err
        }
    } else {
        // unregister secret store
        err := provider.UnregisterStore(secretStore)
        if err != nil {
            log.Error(err, "unable to unregister SecretStore")
            return ctrl.Result{}, err
        }

        err = r.Client.Delete(ctx, secretStore)
        if err != nil {
            log.Error(err, "unable to delete SecretStore")
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

func (r *SecretStoreReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&apis.SecretStore{}).
        Complete(r)
}

// ExternalSecretReconciler reconciles ExternalSecret objects
type ExternalSecretReconciler struct {
    Client client.Client
    Log    logr.Logger
    // Add secret provider here
}

func (r *ExternalSecretReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("ExternalSecret", req.NamespacedName)
    log.Info(fmt.Sprintf("Reconciling External Secret : %s/%s", req.NamespacedName.Namespace, req.NamespacedName.Name))

    // Fetch the ExternalSecret instance
    externalSecret := &externalv1alpha1.ExternalSecret{}
    if err := r.Client.Get(ctx, req.NamespacedName, externalSecret); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        log.Error(err, "unable to fetch ExternalSecret")
        return ctrl.Result{}, err
    }

    if externalSecret.ObjectMeta.DeletionTimestamp.IsZero() {
        // convert remote secret to k8s secret
        remoteSecret, err := provider.GetSecret(*externalSecret, r.Provider())
        if err != nil {
            log.Error(err, "unable to get secret from provider")
            return ctrl.Result{}, err
        }

        data := map[string][]byte{}
        for k, v := range remoteSecret.Data {
            data[k] = []byte(v)
        }

        k8sSecret := &framework.K8sSecret{
            TypeMeta: framework.TypeMetaSecret,
            ObjectMeta: framework.ObjectMeta{
                Namespace: externalSecret.Namespace,
                Name:      externalSecret.Spec.Target.Name,
                Labels:    externalSecret.Spec.Target.Labels,
            },
            Data:      data,
            StringData: remoteSecret.StringData,
            Type:        utils.SecretType(externalSecret.Spec.Target.Type),
        }

        // Set external secret controller reference
        if err := controllerutil.SetControllerReference(externalSecret, k8sSecret, r.Scheme); err != nil {
            log.Error(err, "unable to set controller reference")
            return ctrl.Result{}, err
        }

        // Check if the K8s secret already exists; create it if it doesn't
        k8sSecretObj := &corev1.Secret{}
        err = r.Client.Get(ctx, client.ObjectKeyFromObject(k8sSecret), k8sSecretObj)
        if err != nil {
            if errors.IsNotFound(err) {
                if err := r.Client.Create(ctx, k8sSecret); err != nil {
                    log.Error(err, "unable to create k8s secret")
                    return ctrl.Result{}, err
                }
            } else {
                log.Error(err, "unable to check k8s secret")
                return ctrl.Result{}, err
            }
        } else {
            // Check if the K8s secret needs updating
            if !cmp.Equal(k8sSecretObj.StringData, k8sSecret.StringData) || !cmp.Equal(k8sSecretObj.Data, k8sSecret.Data) {
                k8sSecretObj.StringData = k8sSecret.StringData
                k8sSecretObj.Data = k8sSecret.Data

                if err = r.Client.Update(ctx, k8sSecretObj); err != nil {
                    log.Error(err, "unable to update k8s secret")
                    return ctrl.Result{}, err
                }
            }
        }
    } else {
        // delete k8s secret
        k8sSecret := &corev1.Secret{
            ObjectMeta: framework.ObjectMeta{
                Namespace: externalSecret.Namespace,
                Name:      externalSecret.Spec.Target.Name,
            },
        }
        if err := r.Client.Delete(ctx, k8sSecret); err != nil {
            log.Error(err, "unable to delete k8s secret")
            return ctrl.Result{}, err
        }

        // delete ExternalSecret
        if err := r.Client.Delete(ctx, externalSecret); err != nil {
            log.Error(err, "unable to delete ExternalSecret")
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

func (r *ExternalSecretReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&externalv1alpha1.ExternalSecret{}).
        WithEventFilter(predicate.GenerationChangedPredicate{}).
        Watches(
            &source.Kind{Type: &apis.SecretStore{}},
            &handler.EnqueueRequestsFromMapFunc{
                ToRequests: provider.EnqeueRequestForAllExternalSecrets,
            },
        ).
        Complete(r)
}

This implementation registers and un-registers the SecretStore instance by invoking the provider.RegisterStore() and provider.UnregisterStore() respectively.

Additionally, the implementation converts the remote secret to a Kubernetes secret and updates the Kubernetes secret based on the ExternalSecret spec. Deletion of the ExternalSecret results in the deletion of the corresponding Kubernetes secret.

Note that this implementation uses the provider.Provider and the provider.GetSecret functions which are placeholders for the actual implementation based on the external secret provider.

I hope this helps! Let me know if you have any additional questions.