search

Trow-Registry / trow

Container Registry and Image Management for Kubernetes Clusters
https://trow.io
Apache License 2.0
914 stars 102 forks source link

Trow Installation issue #263

Closed azaadshatru closed 3 years ago

azaadshatru commented 3 years ago

Hi, I am trying to install Trow on my bare metal kubernetes cluster(1 master/4 nodes) running on RHEL8 Vms. Although installation went successfull, I am not able to make the trow registry up and running following are the details and error messages: Please help.

@@@@@@ Installation @@@@@@@@@@@@@@

./install.sh
Trow AutoInstaller for Kubernetes
=================================

This installer assumes kubectl is configured to point to the cluster you want to
install Trow on and that your user has cluster-admin rights.

This installer will perform the following steps:

  - Create a ServiceAccount and associated Roles for Trow
  - Create a Kubernetes Service and Deployment
  - Request and sign a TLS certificate for Trow from the cluster CA
  - Copy the public certificate to all nodes in the cluster
  - Copy the public certificate to this machine (optional)
  - Register a ValidatingAdmissionWebhook (optional)

If you're running on GKE, you may first need to give your user cluster-admin
rights:

  $ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/                           account)

Also make sure port 32001 is open on the firewall so clients can connect.
If you're running on the Google cloud, the following should work:

  $ gcloud compute firewall-rules create trow --allow tcp:32001 --project <project name>

This script will install Trow to the kube-public namespace.
To choose a different namespace run:
  $ ./install.sh <my-namespace>

Do you want to continue? (y/n) y
Installing Trow in namespace: kube-public

Starting Kubernetes Resources
serviceaccount/trow unchanged
role.rbac.authorization.k8s.io/trow unchanged
clusterrole.rbac.authorization.k8s.io/trow unchanged
rolebinding.rbac.authorization.k8s.io/trow unchanged
clusterrolebinding.rbac.authorization.k8s.io/trow unchanged
deployment.apps/trow-deploy unchanged
service/trow unchanged

Approving certificate. This may take some time.

Saving cluster certficate as trow-ca-cert
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIxMDYyMjA3NTIwMloXDTMxMDYyMDA3NTIwMlowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL1X
KH86r+656frS6qcccvMKYMW+jqif5dBqzIV60g8m9F9XCEngZBUKYadEl9dwG12+
i4nKSJEDQXzAvGjqaG9GCkULqGkt4bxAHdW8N5N5yzqODwx+kDDHYjf4YnzuRj5Z
zRHubJBRz36Y/fmA09zrS0+qSdp1gehP/7fFtTr5hryiTVBIulBRk+k6mPGNenj+
w1jYtyDiOLy7wzALpMVhgl6goYhXJa5sdQEG5NpWLEBxYZVP0OeG4IvlyX1zuOA1
vxp0Vi1PVWF3OcDhHVoPhExyk/pYL4/biBtUQq1gFtkNsr8ScP1/jKUuHKjI8OkQ
D9OvlURZljJAcIwf9O0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAJlsnP/vZr+NGNNjf9WiFEZ2XZra
8ECTwZJbNUlEj6hSILpAA0beKPzEhPKyjVyKtdSaAH4HV2oka+yn5XVy5kdjz197
Qc4Kg1Ys1+mHKt0hjYWDw/mmtJ9i2lb0hex2hrAc/zlYA1FYYGArWBKU0Yl2ttSY
Gcozm8GRY6A2rupidFdEVJGodSOG+3ixq+e6bzT4Jager2JrD90DHRzWfOmdDNIn
Auj8uILeunai5hjZ/+of+9jRRX4CvCxpBCRtSqv3kj7Ln3L8Ra3Zep9FDS0IZUuG
rpWyDs9IQwblisLYXDAa9VCSSrM/D/ccH4Z3qFuuY3OGP5O8d4jiOBQM6xQ=
-----END CERTIFICATE-----
configmap/trow-ca-cert created

Copying certs to nodes
job.batch "copy-certs-11109b06-70ba-4943-bc41-715b039e20e4" deleted
job.batch "copy-certs-8c2f88ec-3dbf-4e01-aa14-43048ae3a77e" deleted
job.batch "copy-certs-b5cc1670-237b-48e0-bd44-b67c291a486a" deleted
job.batch "copy-certs-dde5eab6-b10f-4a96-a5d4-51d5d6a02023" deleted
job.batch "copy-certs-fd207e09-2c24-4f03-be38-cc4d1fab466d" deleted
job.batch/copy-certs-dde5eab6-b10f-4a96-a5d4-51d5d6a02023 created
job.batch/copy-certs-11109b06-70ba-4943-bc41-715b039e20e4 created
job.batch/copy-certs-8c2f88ec-3dbf-4e01-aa14-43048ae3a77e created
job.batch/copy-certs-fd207e09-2c24-4f03-be38-cc4d1fab466d created
job.batch/copy-certs-b5cc1670-237b-48e0-bd44-b67c291a486a created

Do you wish to install certs on this host and configure /etc/hosts to allow access from this machine? (y/n) y

Copying cert into Docker
This requires sudo privileges
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIxMDYyMjA3NTIwMloXDTMxMDYyMDA3NTIwMlowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL1X
KH86r+656frS6qcccvMKYMW+jqif5dBqzIV60g8m9F9XCEngZBUKYadEl9dwG12+
i4nKSJEDQXzAvGjqaG9GCkULqGkt4bxAHdW8N5N5yzqODwx+kDDHYjf4YnzuRj5Z
zRHubJBRz36Y/fmA09zrS0+qSdp1gehP/7fFtTr5hryiTVBIulBRk+k6mPGNenj+
w1jYtyDiOLy7wzALpMVhgl6goYhXJa5sdQEG5NpWLEBxYZVP0OeG4IvlyX1zuOA1
vxp0Vi1PVWF3OcDhHVoPhExyk/pYL4/biBtUQq1gFtkNsr8ScP1/jKUuHKjI8OkQ
D9OvlURZljJAcIwf9O0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAJlsnP/vZr+NGNNjf9WiFEZ2XZra
8ECTwZJbNUlEj6hSILpAA0beKPzEhPKyjVyKtdSaAH4HV2oka+yn5XVy5kdjz197
Qc4Kg1Ys1+mHKt0hjYWDw/mmtJ9i2lb0hex2hrAc/zlYA1FYYGArWBKU0Yl2ttSY
Gcozm8GRY6A2rupidFdEVJGodSOG+3ixq+e6bzT4Jager2JrD90DHRzWfOmdDNIn
Auj8uILeunai5hjZ/+of+9jRRX4CvCxpBCRtSqv3kj7Ln3L8Ra3Zep9FDS0IZUuG
rpWyDs9IQwblisLYXDAa9VCSSrM/D/ccH4Z3qFuuY3OGP5O8d4jiOBQM6xQ=
-----END CERTIFICATE-----
Successfully copied cert
Adding entry to /etc/hosts for trow.kube-public

No external IP listed in "kubectl get nodes -o wide"
Trying minikube
Not minikube.
Trying internal IP which may work for local clusters e.g. microk8s

Exposing registry via /etc/hosts
This requires sudo privileges
445
445
172.27.187.183 trow.kube-public # added for trow registry

Successfully configured localhost

Do you want to configure Trow as a validation webhook (NB this will stop external images from being deployed to the cluster)?                            (y/n) y
Setting up trow as a validating webhook
WARNING: This will limit what images can run in your cluster
By default, only images in Trow and official Kubernetes images will be
allowed

validatingwebhookconfiguration.admissionregistration.k8s.io/trow-validator created

@@@@@@ Logs @@@@@@@@@@@@@@@@@
kubectl describe pod trow-deploy-7764957445-4zdvd -n kube-public
Name:         trow-deploy-7764957445-4zdvd
Namespace:    kube-public
Priority:     0
Node:         abc.xyz.com/172.27.186.226
Start Time:   Thu, 08 Jul 2021 20:10:10 +0530
Labels:       app=trow
              pod-template-hash=7764957445
Annotations:  cni.projectcalico.org/podIP: 10.244.70.70/32
              cni.projectcalico.org/podIPs: 10.244.70.70/32
Status:       Pending
IP:           10.244.70.70
IPs:
  IP:           10.244.70.70
Controlled By:  ReplicaSet/trow-deploy-7764957445
Init Containers:
  trow-init:
    Container ID:  docker://afbb4879b8c516bcdda27de7c4d6092a75080548a27ef7b390d83a880f6314d0
    Image:         containersol/trow:init
    Image ID:      docker-pullable://containersol/trow@sha256:ec641dda41d0981cdba2799377c00dc1f852ad14539940c1ab3ba58cd3752fa6
    Port:          <none>
    Host Port:     <none>
    Command:
      /init.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    9
      Started:      Fri, 09 Jul 2021 09:06:20 +0530
      Finished:     Fri, 09 Jul 2021 09:06:35 +0530
    Ready:          False
    Restart Count:  148
    Environment:
      POD_NAME:       trow-deploy-7764957445-4zdvd (v1:metadata.name)
      POD_NAMESPACE:  kube-public (v1:metadata.namespace)
      POD_IP:          (v1:status.podIP)
    Mounts:
      /data/trow/certs from cert-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from trow-token-ghnf6 (ro)
Containers:
  trow-pod:
    Container ID:
    Image:         amouat/trow-test
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      -n
      trow:32001 trow.kube-public:32001
      -c
      /data/trow/certs/domain.crt
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data/trow/certs from cert-vol (rw)
      /data/trow/data from data-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from trow-token-ghnf6 (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  cert-vol:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  data-vol:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  trow-token-ghnf6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  trow-token-ghnf6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                     From     Message
  ----     ------            ----                    ----     -------
  Warning  BackOff           27m (x3260 over 12h)    kubelet  Back-off restarting failed container
  Warning  DNSConfigForming  2m33s (x3670 over 12h)  kubelet  Search Line limits were exceeded, some search paths have been omitted, the applied search line is: kube-public.svc.cluster.local svc.cluster.local cluster.local apac.xyz.com na.xyz.com emea.xyz.com
[xyz@abc1 quick-install]$ kubectl logs trow-deploy-7764957445-4zdvd -n kube-public
Error from server (BadRequest): container "trow-pod" in pod "trow-deploy-7764957445-4zdvd" is waiting to start: PodInitializing
[xyz@abc1 quick-install]$ kubectl get pods -n kube-public -o wide --watch
NAME                                                    READY   STATUS                  RESTARTS   AGE   IP             NODE                     NOMINATED NODE   READINESS GATES
copy-certs-11109b06-70ba-4943-bc41-715b039e20e4-9ndr2   0/1     Completed               0          12h   10.244.73.4    abc2.xyz.com   <none>           <none>
copy-certs-8c2f88ec-3dbf-4e01-aa14-43048ae3a77e-fq2qx   0/1     Completed               0          12h   10.244.69.8    abc3.xyz.com   <none>           <none>
copy-certs-b5cc1670-237b-48e0-bd44-b67c291a486a-cctz8   0/1     Completed               0          12h   10.244.73.70   abc4.xyz.com   <none>           <none>
copy-certs-dde5eab6-b10f-4a96-a5d4-51d5d6a02023-g7qss   0/1     Pending                 0          12h   <none>         <none>                   <none>           <none>
copy-certs-fd207e09-2c24-4f03-be38-cc4d1fab466d-l25kc   0/1     Completed               0          12h   10.244.70.72   abc.xyz.com   <none>           <none>
trow-deploy-7764957445-4zdvd                            0/1     Init:CrashLoopBackOff   148        12h   10.244.70.70   abc.xyz.com   <none>           <none>
amouat commented 3 years ago

Sorry, I only just saw this message. I'll take look next week. Apologies for the delay.

amouat commented 3 years ago

Can you tell me the version of k8s you're using and the container runtime (run kubectl version and kubectl get nodes -o wide)?

I also need the logs from the trow pod. The command will be something like:

kubectl logs -n kube-public trow-deploy-7764957445-4zdvd -c init

Whilst the quick install is fine for home labs, if this is intended to be something long-running, I would look into trying to get cert-manager running and using the standard install methods (Helm or Kustomize). (Either way, I'd still like to figure out what's happening here).

azaadshatru commented 3 years ago

Hi Adrien, Thanks for looking into the issue. I was working on something for home lab when encountered this issue. As of now, I have to move on and I used some other registry to continue. I think you can close this track.

Best Regards.

Shatrughan Saxena +91-9960728251

From: Adrian Mouat @.> Sent: Friday, July 30, 2021 2:48 PM To: ContainerSolutions/trow @.> Cc: Shatrughan Saxena @.>; Author @.> Subject: Re: [ContainerSolutions/trow] Trow Installation issue (#263)

EXTERNAL

Can you tell me the version of k8s you're using and the container runtime (run kubectl version and kubectl get nodes -o wide)?

I also need the logs from the trow pod. The command will be something like:

kubectl logs -n kube-public trow-deploy-7764957445-4zdvd -c init

Whilst the quick install is fine for home labs, if this is intended to be something long-running, I would look into trying to get cert-manager running and using the standard install methods (Helm or Kustomize)

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FContainerSolutions%2Ftrow%2Fissues%2F263%23issuecomment-889760271&data=04%7C01%7Cshatrughan.saxena%40sas.com%7Cd1bd99aa3ab04c02320708d9533af9bd%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637632335056783344%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5zf%2FKUOBzS%2FWzTmT3Xea23qvaNkUL%2FE0YEhwUW1yt90%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMNAHVXELFOZKMTT3M6MWN3T2JU5ZANCNFSM5AB7YFRQ&data=04%7C01%7Cshatrughan.saxena%40sas.com%7Cd1bd99aa3ab04c02320708d9533af9bd%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637632335056793307%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vbpyM%2FS%2FEdpv4HPLQS38UvbA6dN17aaZshSisayMrlA%3D&reserved=0.

amouat commented 3 years ago

Ok, thanks for the update.