Support eksctl profiles

chanwit commented 5 years ago

As a user,

I would like to reuse profiles from eksctl and apply to my local cluster.

So that I can have the similar cluster functionalities and help me validate the behavior of my EKS cluster.

Local eksctl-like experience: we could use Firekube to replicate eksctl + AWS behavior by having the same set of eksctl profiles running on top of Firekube. This will be enabled by implementing --profile option for wksctl.

Here's the comment from @mflendrich on UX:

In this proposal enable becomes a command group that has one subcommand: profile. disable becomes another command group with one subcommand profile. This solution has an issue that enable/disable don't wrap a bounded context in the problem domain, so it might be better to proceed with wksctl profile as a command group with subcommands enable/disable: that would mean wksctl profile enable, wksctl profile disable

chanwit commented 5 years ago

How the profile implementation fits within GitOps? Currently, GitOps with wksctl is working with a single repository. wksctl profile enable app-dev clones the app-dev repo into the main repo as a directory. With this, users proceed with the same GitOps workflow, says, commit and push.

What's role of Flux? Flux still plays the main role here. It monitors the remote repo after users applied the wksctl profile enable command and git push profile manifests. It then recursively scans the repo, including each profile directories per this behaviour.

https://docs.fluxcd.io/en/stable/requirements.html?highlight=recursive

Then apply profile's manifests to the cluster.

What's the cleanup process? To clean up profiles, there will be two possible ways.

wksctl profile disable app-dev it will delete the profile directory for profile app-dev.
manually delete the profile directories, commit and push.

palemtnrider commented 5 years ago

Is the thought being the user would run wksctl profile before or after wksctl apply? Seems like it could be either?

chanwit commented 5 years ago

Yes, either.

palemtnrider commented 5 years ago

would wksctl perform any of the commit/push operations?

palemtnrider commented 5 years ago

Does the user pass the --git-url argument or does this command assume it's operating within a local clone?

chanwit commented 5 years ago

would wksctl perform any of the commit/push operations?

I don't think so. Commit and push should leave for the user to decide.

chanwit commented 5 years ago

Does the user pass the --git-url argument or does this command assume it's operating within a local clone?

Depends on what you do mean by --git-url?

If it's for the main repository, which will be monitored by Flux, then the answer is no.
If it's the profile repository, which will be immediately downloaded by profile enable command, then yes. <-- I'm guessing this is what you meant by operating within a local clone.

To be a bit more elaborated,

A user initializes the cluster: wksctl apply --git-url=git@github.com:chanwit/wksctl-quickstart-firekube

at this point, the user got a local repository

The user then enable the app-dev profile: wksctl profile enable --git-url=github.com/weaveworks/eks-quickstart-app-dev or might be: wksctl profile enable github.com/weaveworks/eks-quickstart-app-dev

at this point, the user operate the profile command within the local repo.

chanwit commented 5 years ago

This is the current PoC: Profile app-dev is hard-coded to github.com/weaveworks/eks-quickstart-app-dev. (Similar to the eksctl behaviour)

$ ~/.wks/bin/wksctl profile enable \
  --repository=app-dev \
  --revision=dafeaef2c9bf77f948d6685a4e2031c862fcfaf3
INFO[2019-09-30T01:31:54+07:00] Cloning into "profiles/github.com/weaveworks/eks-quickstart-app-dev" ... 
INFO[2019-09-30T01:32:03+07:00] Cleanup .git ...                             
$ tree profiles 
profiles
└── github.com
    └── weaveworks
        └── eks-quickstart-app-dev
            ├── amazon-cloudwatch
            │   ├── cloudwatch-agent-configmap.yaml.tmpl
            │   ├── cloudwatch-agent-daemonset.yaml
            │   ├── cloudwatch-agent-rbac.yaml
            │   ├── fluentd-configmap-cluster-info.yaml.tmpl
            │   ├── fluentd-configmap-fluentd-config.yaml
            │   ├── fluentd-daemonset.yaml
            │   └── fluentd-rbac.yaml
            ├── demo
            │   └── helm-release.yaml
            ├── kubernetes-dashboard
            │   ├── dashboard-metrics-scraper-deployment.yaml
            │   ├── dashboard-metrics-scraper-service.yaml
            │   ├── kubernetes-dashboard-configmap.yaml
            │   ├── kubernetes-dashboard-deployment.yaml
            │   ├── kubernetes-dashboard-rbac.yaml
            │   ├── kubernetes-dashboard-secrets.yaml
            │   └── kubernetes-dashboard-service.yaml
            ├── kube-system
            │   ├── alb-ingress-controller-deployment.yaml.tmpl
            │   ├── alb-ingress-controller-rbac.yaml
            │   ├── cluster-autoscaler-deployment.yaml.tmpl
            │   └── cluster-autoscaler-rbac.yaml
            ├── LICENSE
            ├── monitoring
            │   ├── metrics-server.yaml
            │   └── prometheus-operator.yaml
            ├── namespaces
            │   ├── amazon-cloudwatch.yaml
            │   ├── demo.yaml
            │   ├── kubernetes-dashboard.yaml
            │   └── monitoring.yaml
            └── README.md

9 directories, 27 files

palemtnrider commented 5 years ago

If we are cloning into an existing repo, does this create a git submodule? If so, does will flux work correctly?

palemtnrider commented 5 years ago

How do we prohibit a user from pushing back to the origin (since it comes from weaveworks/eks-quickstart)?

chanwit commented 5 years ago

If we are cloning into an existing repo, does this create a git submodule? If so, does will flux work correctly?

I just fixed the behaviour of profile enable to delete .git directory after cloning. So, a profile won't appear as a git submodule, but plain set of files.

How do we prohibit a user from pushing back to the origin (since it comes from weaveworks/eks-quickstart)?

After delete .git from the cloned profile, a user won't be able to push back to the upstream.

chanwit commented 5 years ago

The only problem at the moment is that when we trying to run wksctl profile enable app-dev again, the command stops and returns error.

But the behaviour should be updating the profile to the specified version. Anyway, this might cause another problem if the user already made changes to the profile codes.

chanwit commented 5 years ago

After re-iterate the idea, I think we need auto-commit and push as the default behaviour. Then we need --no-commit flag for profile enable.

which means wksctl profile enable app-dev will auto commit and push to make the UX good.

wksctl profile enable --no-commit app-dev will leave users to commit and push manually.

chanwit commented 5 years ago

Here's the current behaviour with auto commit & push. The profile used in this demo is a simple one from: https://github.com/chanwit/podinfo-profile

$ ~/.wks/bin/wksctl profile enable \
  --repository=https://github.com/chanwit/podinfo-profile
INFO[2019-10-01T17:16:11+07:00] Cloning into "profiles/github.com/chanwit/podinfo-profile" ... 
Cloning into 'profiles/github.com/chanwit/podinfo-profile'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 0), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
INFO[2019-10-01T17:16:13+07:00] Removing .git directory ...                  
INFO[2019-10-01T17:16:13+07:00] Adding profile https://github.com/chanwit/podinfo-profile to the local repository ... 
INFO[2019-10-01T17:16:13+07:00] Adding clonePath profiles/github.com/chanwit/podinfo-profile to the local repository ... 
INFO[2019-10-01T17:16:13+07:00] Added profile from https://github.com/chanwit/podinfo-profile ... 
INFO[2019-10-01T17:16:13+07:00] Committing the profile ...                   
INFO[2019-10-01T17:16:13+07:00] Use default user name and password ...       
[master 8a6de13] Enable profile: https://github.com/chanwit/podinfo-profile
 2 files changed, 124 insertions(+)
 create mode 100644 profiles/github.com/chanwit/podinfo-profile/namespaces/podinfo_ns.yaml
 create mode 100644 profiles/github.com/chanwit/podinfo-profile/workloads/podinfo.yaml
INFO[2019-10-01T17:16:13+07:00] Committed the profile ...                    
INFO[2019-10-01T17:16:13+07:00] Pushing to the remote ...                    
Counting objects: 10, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (10/10), 1.50 KiB | 767.00 KiB/s, done.
Total 10 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To github.com:chanwit/wks-quickstart-firekube.git
   8b5e49e..8a6de13  master -> master
INFO[2019-10-01T17:16:18+07:00] Pushed successfully.                         
$ kubectl get ns   
NAME                   STATUS   AGE
default                Active   22m
kube-node-lease        Active   22m
kube-public            Active   22m
kube-system            Active   22m
podinfo-profile-demo   Active   1s
weavek8sops            Active   21m
$ kubectl get all -n podinfo-profile-demo
NAME                           READY   STATUS    RESTARTS   AGE
pod/podinfo-5fd5f87f59-4hvxk   1/1     Running   0          22s
pod/podinfo-5fd5f87f59-8zw2r   1/1     Running   0          38s

NAME              TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.100.52.8   <none>        9898/TCP,9999/TCP   39s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/podinfo   2/2     2            2           39s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/podinfo-5fd5f87f59   2         2         2       38s

NAME                                          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/podinfo   Deployment/podinfo   <unknown>/99%   2         4         2          38s

weaveworks / wksctl

Support eksctl profiles #82