Enabled operator installation for karmor

rootxrishabh commented 7 months ago

Purpose: This PR shifts karmor install and karmor uninstall to use kubearmor operator using helm.

Does this PR introduce a breaking change? Yes

Aryan-sharma11 commented 6 months ago

Snyk issues remediation:-

Upgrade github.com/opencontainers/runc/libcontainer to version 1.1.12 or higher. ( and all pkgs related to it)
Upgrade github.com/lestrrat-go/jwx/v2/jws to version 2.0.19 or higher.
Upgrade go.etcd.io/etcd/v3/server/lease to version 3.4.29, 3.5.11 or higher
Upgrade go.etcd.io/etcd/v3/server/etcdserver to version 3.4.29, 3.5.11 or higher.

There are still some pkgs whose fixes are pushed to the main branch but not published yet.

github.com/urfave/negroni/v2 introduced though github.com/sigstore/timestamp-authority@v1.2.0

daemon1024 commented 6 months ago

Fixes https://github.com/kubearmor/KubeArmor/issues/1256

daemon1024 commented 6 months ago

karmor install --save failing if no kubernetes cluster is configured We do not mandate K8s cluster if we only need to save

daemon1024 commented 6 months ago

Registry flag is not working

./karmor install -r "ttl.sh" --save

Expected - config images and operator image to be prepended with registry info Actual - No change in config

daemon1024 commented 6 months ago

./karmor install --tag=v1.2.1 --save

Not working as expected

Expected - config images and operator image to have their tags changed Actual - No change in config

daemon1024 commented 6 months ago

I couldn't help but notice, even if I am just using save. It takes a lot of time to generate things. Is it a drawback of using helm client?

rootxrishabh commented 6 months ago

I couldn't help but notice, even if I am just using save. It takes a lot of time to generate things. Is it a drawback of using helm client?

I tried printing manifests without running the client and it returns a null pointer. So, the install/upgrade client had to dryrun in order to get the manifests. Taking a look at other points :eyes:

rootxrishabh commented 6 months ago

Have we verified this behaviour with a legacy karmor install? Please include screenshots

`rootxrishabh@rootxrishabh:~/kubearmor-client$ ./karmor install --legacy=true 😄 Environment : k3s
🔥 CRD kubearmorpolicies.security.kubearmor.com
ℹ️ CRD kubearmorpolicies.security.kubearmor.com already exists
🔥 CRD kubearmorhostpolicies.security.kubearmor.com
ℹ️ CRD kubearmorhostpolicies.security.kubearmor.com already exists
💫 Service Account
💫 Service Account
⚙️ Cluster Role
ℹ️ Cluster Role already exists
⚙️ Cluster Role Bindings
ℹ️ Cluster Role Bindings already exists
⚙️ KubeArmor Relay Roles
🛡 KubeArmor Relay Service
🛰 KubeArmor Relay Deployment
🛡 KubeArmor DaemonSet - Init kubearmor/kubearmor-init:stable, Container kubearmor/kubearmor:stable -gRPC=32767
ℹ️ KubeArmor DaemonSet already exists
🛡 KubeArmor Controller TLS certificates
💫 KubeArmor Controller Service Account
⚙️ KubeArmor Controller Roles
🚀 KubeArmor Controller Deployment
🚀 KubeArmor Controller Metrics Service
🚀 KubeArmor Controller Webhook Service
🥳 Done Installing KubeArmor
🤩 KubeArmor Controller Mutation Admission Registration
ℹ️ KubeArmor Controller Mutation Admission Registration already exists 88% 🚀 KubeArmor ConfigMap Creation 88% 😋 Checking if KubeArmor pods are running...▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇] 111.76% 🥳 Done Checking , ALL Services are running!
⌚️ Execution Time : 13.858858021s

🔧 Verifying KubeArmor functionality (this may take upto a minute)...

    🛡️       Your Cluster is Armored Up!

rootxrishabh@rootxrishabh:~/kubearmor-client$ ./karmor uninstall ❌ KubeArmor is either not installed, or the specified namespace is incorrect. ℹ️ Please ensure you have installed KubeArmor, and check that you are specifying the correct namespace. ℹ️ Attempting legacy uninstallation. 🗑️ Mutation Admission Registration 🗑️ KubeArmor Services ❌ Service: kubearmor removed ❌ Service: kubearmor-controller-metrics-service removed 💨 Service Accounts ❌ ServiceAccount kubearmor removed ❌ ServiceAccount kubearmor-relay removed ❌ ServiceAccount kubearmor-controller removed 💨 Cluster Roles ❌ ClusterRole kubearmor-snitch removed ❌ ClusterRole kubearmor-clusterrole removed ❌ ClusterRole kubearmor-relay-clusterrole removed ❌ ClusterRole kubearmor-controller-proxy-role removed ❌ ClusterRole kubearmor-controller-clusterrole removed 💨 Cluster Role Bindings ❌ ClusterRoleBinding kubearmor-clusterrolebinding removed ❌ ClusterRoleBinding kubearmor-relay-clusterrolebinding removed ❌ ClusterRoleBinding kubearmor-controller-clusterrolebinding removed ❌ ClusterRoleBinding kubearmor-controller-proxy-rolebinding removed ❌ ClusterRoleBinding kubearmor-snitch-binding removed 🧹 Roles ❌ Role kubearmor-controller-leader-election-role removed 🧹 RoleBindings ❌ RoleBinding kubearmor-controller-leader-election-rolebinding removed 👻 KubeArmor Controller TLS certificates ❌ KubeArmor Controller TLS certificate kubearmor-controller-webhook-server-cert removed 👻 KubeArmor ConfigMap ❌ ConfigMap kubearmor-config removed 👻 KubeArmor DaemonSet ❌ KubeArmor DaemonSet kubearmor removed 👻 KubeArmor Deployments ❌ KubeArmor Deployment kubearmor-relay removed ❌ KubeArmor Deployment kubearmor-controller removed 🔄 Checking if KubeArmor pods are stopped... 🔴 Done Checking; all services are stopped!
⌚️ Termination Time: 9.787902842s ` Yes

daemon1024 commented 6 months ago

Installation Process in general is smooooth. Great Work.

But this does not look cohesive.

the helm logs and emoji logs are not streamlines, We should suppress helm logs and manually log based on the output maybe. Not sure what's best here.
We are spamming Deployment is not ready messages
We need to type something when KubeArmor Daemonset is deploying, Because it takes a while, including messaging around expected time, inform that snitch is running

daemon1024 commented 6 months ago

karmor uninstall

This is not working as expected I did the standard helm install But it is saying it cannot detect it It fellback to legacy uninstallation, but my helm package still exists

rootxrishabh commented 6 months ago

Installation Process in general is smooooth. Great Work.

But this does not look cohesive.

* the helm logs and emoji logs are not streamlines, We should suppress helm logs and manually log based on the output maybe. Not sure what's best here.

* We are spamming `Deployment is not ready` messages

* We need to type something when KubeArmor Daemonset is deploying, Because it takes a while, including messaging around expected time, inform that snitch is running

I think we should handle logs ourselves and suppress helm logs. We could manually output logs for deployments etc. I will add deployment message for snitch and include an expected time logs.

rootxrishabh commented 6 months ago

karmor uninstall
This is not working as expected I did the standard helm install But it is saying it cannot detect it It fellback to legacy uninstallation, but my helm package still exists

Yes, I was thinking of discussing this earlier. If we remove the default namespace in uninstall and don't specify namespace for example "kubearmor" then our current helm implementation doesn't automatically detect the installed namespace. I guess implementing auto detection of namespace is required?

daemon1024 commented 6 months ago

I guess implementing auto detection of namespace is required?

Yup. We can list packages and go through that ig.

rootxrishabh commented 6 months ago

I don't think the first log is looking nice, is it something we need to keep?

It is not necessary, just denoting the initiation of helm client. Should we suppress logs for save flag?

daemon1024 commented 6 months ago

I think it will automatically be handled when we have the entire logging figured out for helm.

rootxrishabh commented 6 months ago

./karmor install --tag=v1.2.1 --save
Not working as expected

Expected - config images and operator image to have their tags changed Actual - No change in config

We currently dont have --save watching tag and registry flags. I will do it :+1:

daemon1024 commented 6 months ago

We currently dont have --save watching tag and registry flags. I will do it 👍

That's concerning. Ideal case it should be not be 2 different approaches. We finalise on the config then decide whether to save it or install it.

rootxrishabh commented 6 months ago

We currently dont have --save watching tag and registry flags. I will do it 👍

That's concerning. Ideal case it should be not be 2 different approaches. We finalise on the config then decide whether to save it or install it.

Got it. I will make sure we are handling all flags first.

rootxrishabh commented 6 months ago

I couldn't help but notice, even if I am just using save. It takes a lot of time to generate things. Is it a drawback of using helm client?

Update: I fixed --save not running for clusterless environments and now the process is running client side only, executes without helm logs and is quick.

rootxrishabh commented 6 months ago

Here's a demo of the functionality - kubearmorgif

daemon1024 commented 6 months ago

We should be updating the operator image as well when we use tag/flags/registry

operator image is not changing with any flags. We might need to impement a new flag to update operator image. I believe the helm chart accepts updates to operator image as well

rootxrishabh commented 6 months ago

We can now change imagePullPolicy, tag, image, registry.

DelusionalOptimist commented 6 months ago

Also the uninstall command stays stuck right now because of - https://github.com/kubearmor/KubeArmor/issues/1638

rootxrishabh commented 6 months ago

@DelusionalOptimist can we address the review changes first and then take a look at the controller issue?

DelusionalOptimist commented 6 months ago

@DelusionalOptimist can we address the review changes first and then take a look at the controller issue?

Yeah yeah it's cool. Just mentioned it here as an FYI for reviewers. We can look into it later. This PR is priority.

PrimalPimmy commented 6 months ago

@rootxrishabh the defaultposture flags are not working for me

I used block capabilities as flag but it is still in audit mode

rootxrishabh commented 6 months ago

@PrimalPimmy can you please check the kubearmor-config configmap. Currently posture and visibility settings are being incorrectly displayed by karmor probe for operator installation. Ref

PrimalPimmy commented 6 months ago

@PrimalPimmy can you please check the kubearmor-config configmap. Currently posture and visibility settings are being incorrectly displayed by karmor probe for operator installation.

Still not working

file is not going to audit mode when set?

rootxrishabh commented 6 months ago

I see that the kubearmorconfig is not being recreated. You need to do karmor uninstall --force to remove the CR and then reinstall with the posture settings.

Aryan-sharma11 commented 6 months ago

Snyk issues remediation:-

Upgrade helm.sh/helm/v3/pkg to version 3.14.2 or higher.
Upgrade github.com/opencontainers/runc/libcontainer to version 1.1.12 or higher, introduced through github.com/Microsoft/hcsshim@v0.11.4.
Upgrade github.com/lestrrat-go/jwx/v2/jws to version 2.0.19 or higher, Introduced through github.com/Microsoft/hcsshim@v0.11.4 .
Upgrade go.etcd.io/etcd/v3/server to version 3.4.29, 3.5.11 or higher, Introduced through github.com/google/certificate-transparency-go@v1.1.7.
for github.com/urfave/negroni introduced through github.com/sigstore/timestamp-authority@v1.2.0 › github.com/urfave/negroni@v1.0.0 fix pushed to master branch but not yet published .
no fixed version for helm.sh/helm/v3/cmd/helm.

PrimalPimmy commented 6 months ago

I see that the kubearmorconfig is not being recreated. You need to do karmor uninstall --force to remove the CR and then reinstall with the posture settings.

Is there any way to get the configmap recreated, imo it's a better way. Maybe remove the configmap after uninstalling

rootxrishabh commented 6 months ago

I see that the kubearmorconfig is not being recreated. You need to do karmor uninstall --force to remove the CR and then reinstall with the posture settings.

Is there any way to get the configmap recreated, imo it's a better way. Maybe remove the configmap after uninstalling

The configmap kubearmor-config does get deleted but the operator CR kubearmorconfig-default does not and so for every new installation after this, the CR does not get updated and thus we don't see posture settings supplied through CLI. Inorder to remove the CR(kubearmorconfig-default), we must do karmor uninstall --force and do a fresh installation. Ref

rootxrishabh commented 6 months ago

I see that the kubearmorconfig is not being recreated. You need to do karmor uninstall --force to remove the CR and then reinstall with the posture settings.

Is there any way to get the configmap recreated, imo it's a better way. Maybe remove the configmap after uninstalling

The configmap kubearmor-config does get deleted but the operator CR kubearmorconfig-default does not and so for every new installation after this, the CR does not get updated and thus we don't see posture settings supplied through CLI. Inorder to remove the CR(kubearmorconfig-default), we must do karmor uninstall --force and do a fresh installation. Ref

We can notice it outputs KubeArmorConfig created in a fresh installation.

daemon1024 commented 6 months ago

If I do a kubectl apply -f with the update config, it does update existing config if there are changes. I believe the same behaviour should translate here

daemon1024 commented 6 months ago

We should switch to doing

err:= Resource.Create()
if !strings.Contains(err.Error(), "already exists") {
        err:= Resource.Update()
        return err/nil
}

rootxrishabh commented 6 months ago

If I do a kubectl apply -f with the update config, it does update existing config if there are changes. I believe the same behaviour should translate here

I remember us discussing the idea of karmor update. I think we discussed that we would update all operator CR options separately using the update command, it can be handled in a separate issue. Wdyt?

daemon1024 commented 6 months ago

I remember us discussing the idea of karmor update. I think we discussed that we would update all operator CR options separately using the update command, it can be handled in a separate issue. Wdyt?

This is a different issue. People won't realise that CR is not updated. the update command was meant to patch things as well patch the helm release if needed.

We cannot ask people to run uninstall with force just so that they have a fresh installation.

It's a 2 liner change here so I believe it's worth it to handle here. Wdyt?

rootxrishabh commented 6 months ago

I remember us discussing the idea of karmor update. I think we discussed that we would update all operator CR options separately using the update command, it can be handled in a separate issue. Wdyt?

This is a different issue. People won't realise that CR is not updated. the update command was meant to patch things as well patch the helm release if needed.

We cannot ask people to run uninstall with force just so that they have a fresh installation.

It's a 2 liner change here so I believe it's worth it to handle here. Wdyt?

Agreed. --force uninstallation shouldn't be an intermediatory to update the CR. I will implement the CR updation here.

kubearmor / kubearmor-client

Enabled operator installation for karmor #402