search

Trow-Registry / trow

Container Registry and Image Management for Kubernetes Clusters
https://trow.io
Apache License 2.0
907 stars 101 forks source link

Test Trow on Windows with Quick Install #198

Open amouat opened 3 years ago

amouat commented 3 years ago

Try installing Trow on windows using WSL, minikube and the quick install. I think it will fail due to routing issues, but these should be resolvable (e.g. with hostctl suggestion in #189).

Document any issues.

sashkachan commented 3 years ago

I tested the quick install on WSL2 (Ubuntu 20.04), Docker for desktop (3.0), Docker engine (20.10.0), Kubernetes (v1.19.3)

Seems quick install works out of the box.

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ ./install.sh
Trow AutoInstaller for Kubernetes
=================================

This installer assumes kubectl is configured to point to the cluster you want to
install Trow on and that your user has cluster-admin rights.

This installer will perform the following steps:

  - Create a ServiceAccount and associated Roles for Trow
  - Create a Kubernetes Service and Deployment
  - Request and sign a TLS certificate for Trow from the cluster CA
  - Copy the public certificate to all nodes in the cluster
  - Copy the public certificate to this machine (optional)
  - Register a ValidatingAdmissionWebhook (optional)

If you're running on GKE, you may first need to give your user cluster-admin
rights:

  $ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)

Also make sure port 31000 is open on the firewall so clients can connect.
If you're running on the Google cloud, the following should work:

  $ gcloud compute firewall-rules create trow --allow tcp:31000 --project <project name>

This script will install Trow to the kube-public namespace.
To choose a different namespace run:
  $ ./install.sh <my-namespace>

Do you want to continue? (y/n) y
Installing Trow in namespace: kube-public

Starting Kubernetes Resources
serviceaccount/trow created
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/trow created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/trow created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/trow created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/trow created
deployment.apps/trow-deploy created
service/trow created

Approving certificate. This may take some time.
.........
Saving cluster certficate as trow-ca-cert
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
W0108 11:51:56.399555    2242 helpers.go:553] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/trow-ca-cert created

Copying certs to nodes
job.batch/copy-certs-c09b8cd4-5863-45f0-8267-906917d4c7de created

Do you wish to install certs on this host and configure /etc/hosts to allow access from this machine? (y/n) y

Copying cert into Docker
This requires sudo privileges
[sudo] password for shk:
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
Successfully copied cert
Adding entry to /etc/hosts for trow.kube-public

No external IP listed in "kubectl get nodes -o wide"
Trying minikube
Not minikube.
Trying internal IP which may work for local clusters e.g. microk8s

Exposing registry via /etc/hosts
This requires sudo privileges
543
543
192.168.65.3 trow.kube-public # added for trow registry

Successfully configured localhost

Do you want to configure Trow as a validation webhook (NB this will stop external images from being deployed to the cluster)? (y/n) y
Setting up trow as a validating webhook
WARNING: This will limit what images can run in your cluster
By default, only images in Trow and official Kubernetes images will be
allowed

Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/trow-validator created

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ cat /etc/hosts
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
127.0.1.1       DESKTOP-86TH1IU.localdomain     DESKTOP-86TH1IU

192.168.178.25  host.docker.internal
192.168.178.25  gateway.docker.internal
127.0.0.1       kubernetes.docker.internal

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.65.3 trow.kube-public # added for trow registry
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker pull nginx:alpine
alpine: Pulling from library/nginx
801bfaa63ef2: Pull complete
b1242e25d284: Pull complete
7453d3e6b909: Pull complete
07ce7418c4f8: Pull complete
e295e0624aa3: Pull complete
Digest: sha256:c2ce58e024275728b00a554ac25628af25c54782865b3487b11c21cafb7fabda
Status: Downloaded newer image for nginx:alpine
docker.io/library/nginx:alpine
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker tag nginx:alpine trow.kube-public:31000/test/nginx:alpine
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker push trow.kube-public:31000/test/nginx:alpine
The push refers to repository [trow.kube-public:31000/test/nginx]
3633e038dbe3: Pushed
e8f8cd3583be: Pushed
0614f8d14b89: Pushed
029c325415ee: Pushed
777b2c648970: Pushed
alpine: digest: sha256:3730151de47b22325415a895b74381de5ce55c71bfc66f938e88c3fba724c60f size: 1340

Admission webhook seems to work as well.

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker pull nginx:alpine
alpine: Pulling from library/nginx
801bfaa63ef2: Pull complete
b1242e25d284: Pull complete
7453d3e6b909: Pull complete
07ce7418c4f8: Pull complete
e295e0624aa3: Pull complete
Digest: sha256:c2ce58e024275728b00a554ac25628af25c54782865b3487b11c21cafb7fabda
Status: Downloaded newer image for nginx:alpine
docker.io/library/nginx:alpine
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker tag nginx:alpine trow.kube-public:31000/test/nginx:alpine
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ docker push trow.kube-public:31000/test/nginx:alpine
The push refers to repository [trow.kube-public:31000/test/nginx]
3633e038dbe3: Pushed
e8f8cd3583be: Pushed
0614f8d14b89: Pushed
029c325415ee: Pushed
777b2c648970: Pushed
alpine: digest: sha256:3730151de47b22325415a895b74381de5ce55c71bfc66f938e88c3fba724c60f size: 1340

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl create deploy proxy --image=docker.io/nginx
deployment.apps/proxy created
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl get deployment proxy
[...]
Warning  FailedCreate  1s (x12 over 11s)  replicaset-controller  Error creating: admission webhook "validator.trow.io" denied the request: Remote image docker.io/nginx disallowed as not contained in this registry and not in allow list

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl delete deploy proxy
deployment.apps "proxy" deleted
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl create deploy proxy --image=trow.kube-public:31000/test/nginx:alpine
deployment.apps/proxy created
shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl describe rs proxy
Name:           proxy-86594dc79d
Namespace:      default
Selector:       app=proxy,pod-template-hash=86594dc79d
Labels:         app=proxy
[...]
 Events:
 Type    Reason            Age   From                   Message
 ----    ------            ----  ----                   -------
 Normal  SuccessfulCreate  6s    replicaset-controller  Created pod: proxy-86594dc7
sashkachan commented 3 years ago

Running minikube through WSL with --driver=docker works, but the registry is inaccessible from the host. I think the reason since minikube runs in a container, it does not publish the port that trow binds to with Nodeport. So, 31000 is inaccessible, even though the record in /etc/hosts is created correctly (using minikube ip call)

sashkachan commented 3 years ago

Additionally, with validation webhook enabled, trying to create a pod returns this error (granted, the image is not in the registry):

Error from server: admission webhook "validator.trow.io" denied the request: Local image trow.kube-public:31000/test/busybox:latest disallowed as not contained in this registry and not in allow list

Inminikube ssh, pushing the image

docker@minikube:~$ docker push trow.kube-public:31000/test/nginx:latest
The push refers to repository [trow.kube-public:31000/test/nginx]
4eaf0ea085df: Layer already exists
2c7498eef94a: Layer already exists
7d2b207c2679: Layer already exists
5c4e5adc71a8: Layer already exists
87c8a1d8f54f: Layer already exists
latest: digest: sha256:04bfaaf554f4ec4493ea16c45ce93649d905ed9ebdaacfa16a286cadd3fd95fc size: 1222

works as expected, then, on the host, running the image succeeds

shk@DESKTOP-86TH1IU:~/projects/trow/quick-install$ kubectl run --restart=Never nginx --image=trow.kube-public:31000/test/nginx:latest
pod/nginx created