search

kurokobo / awx-on-k3s

An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.
MIT License
560 stars 160 forks source link

Nothing is up on port 443 (or 80) ? - Why ? #3

Closed nashford77 closed 3 years ago

nashford77 commented 3 years ago

root@u500-cube-server:~/awx-on-k3s# kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.43.0.1 443/TCP 29m kube-system kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 29m kube-system metrics-server ClusterIP 10.43.55.221 443/TCP 29m kube-system traefik LoadBalancer 10.43.185.225 192.168.5.104 80:31716/TCP,443:31604/TCP 28m default awx-operator-metrics ClusterIP 10.43.102.194 8383/TCP,8686/TCP 28m awx awx-postgres ClusterIP None 5432/TCP 25m awx awx-service ClusterIP 10.43.251.54 80/TCP 25m

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx get awx,all,ingress,secrets NAME AGE awx.awx.ansible.com/awx 29m

NAME READY STATUS RESTARTS AGE pod/awx-postgres-0 1/1 Running 0 29m pod/awx-59ff55b5b-2czpx 4/4 Running 2 28m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-postgres ClusterIP None 5432/TCP 29m service/awx-service ClusterIP 10.43.251.54 80/TCP 28m

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx 1/1 1 1 28m

NAME DESIRED CURRENT READY AGE replicaset.apps/awx-59ff55b5b 1 1 1 28m

NAME READY AGE statefulset.apps/awx-postgres 1/1 29m

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/awx-ingress awx.tunninet.com 192.168.5.104 80, 443 28m

NAME TYPE DATA AGE secret/awx-admin-password Opaque 1 29m secret/default-token-fwskh kubernetes.io/service-account-token 3 29m secret/awx-postgres-configuration Opaque 6 29m secret/awx-secret-tls kubernetes.io/tls 2 29m secret/awx-app-credentials Opaque 3 28m secret/awx-token-mdtqh kubernetes.io/service-account-token 3 28m secret/awx-secret-key Opaque 1 29m secret/awx-broadcast-websocket Opaque 1 29m

nashford77 commented 3 years ago

On a NC, Telnet, CURL etc - nothing there - also nothing in 'netstat -an' ? nothing else is on 443 or 80 ....

Randy29800 commented 3 years ago

Hi same problem here.

kurokobo commented 3 years ago

@nashford77 @Randy29800 Thanks for reporting,

The K3s (to be accurate, its Traefik) forwards packets using iptables' rules, so it doesn't listen directly on the host's port. It is expected behavior that netstat -an doesn't show any process which listenning on :80 or :443.

If curl <Your External IP, in @nashford77's case 192.168.5.104> returns 404 page not found, it's working fine. You can use your AWX at https://awx.tunninet.com/

$ curl 192.168.0.219
404 page not found

If this returns Connection refused even if you stopped your firewalld, it may be the same as this issue in K3s. You can try the workaround in this comment.

Randy29800 commented 3 years ago

@kurokobo Thanks for you reply. I checked that the static files are not in the folder projects. After the deployment the /data/folder in empty. The folder postgres well contained the database. To check if the folder is correct, i put it in 777. After redeploy come back to 755 but is 0:1000 (instead of 1000:0) so something happen on it. Don't understand why no files are in projects. Any idea? (Sorry for my english)

kurokobo commented 3 years ago

@Randy29800 Seems you're commenting about the issue #2 instead of this issue (#3) ? Could you please move on to #2 or create new one.

nashford77 commented 3 years ago

I tried those random ports as well - they do not respond either. If it relies on IP tables, would forwarding have to be enabled? That's not default of course on Ubuntu. I tried on local house to connect to those random ports without luck

On Thu, Aug 26, 2021, 8:31 AM kurokobo @.***> wrote:

@nashford77 https://github.com/nashford77 @Randy29800 https://github.com/Randy29800 Thanks for reporting,

The K3s (to be accurate, its Traefik) forwards packets using iptables' rules, so it doesn't listen directly on the host's port. It is expected behavior that netstat -an doesn't show any process which listenning on :80 or :443.

If curl <Your External IP, in @nashford77's case 192.168.5.104> returns 404 page not found, it's working fine. You can use your AWX at https://awx.tunninet.com/

$ curl 192.168.0.219 404 page not found

If this returns Connection refused even if you stopped your firewalld, it may be the same as this issue https://github.com/k3s-io/k3s/issues/1414 in K3s. You can try the workaround in this comment https://github.com/k3s-io/k3s/issues/1414#issuecomment-770038893.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kurokobo/awx-on-k3s/issues/3#issuecomment-906366054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSSJU7TL3GJTQZSUUSKINDT6YX2NANCNFSM5C2MTTNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

kurokobo commented 3 years ago

@nashford77 Nope, I mean you should try this workaround, which applies and patches a spec for traefik. Can you tell me the OS and platform? I can't reproduce it in my environment.

nashford77 commented 3 years ago

Ubuntu 20.04 vanilla - nothing fancy on it, was a new build totally

On Thu, Aug 26, 2021, 10:53 AM kurokobo @.***> wrote:

@nashford77 https://github.com/nashford77 Nope, I mean you should try this workaround https://github.com/k3s-io/k3s/issues/1414#issuecomment-770038893, which applies and patches a spec for traefik. Can you tell me the OS and platform? I can't reproduce it in my environment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kurokobo/awx-on-k3s/issues/3#issuecomment-906484069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSSJU6V5UYK3E55XYLHNE3T6ZIQBANCNFSM5C2MTTNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

nashford77 commented 3 years ago

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: kube-system creationTimestamp: "2021-08-26T02:34:45Z" labels: app.kubernetes.io/instance: traefik app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: traefik helm.sh/chart: traefik-9.18.2 name: traefik namespace: kube-system resourceVersion: "33339" uid: 5512d2e3-e339-4821-8de8-5236b73cba6d spec: clusterIP: 10.43.185.225 clusterIPs:

(I have this in the hosts file just to test...)

root@u500-cube-server:~/awx-on-k3s# curl https://awx.tunninet.com --insecure <no response, hitting enter lots>

^C root@u500-cube-server:~/awx-on-k3s# curl http://awx.tunninet.com --insecure <no response, hitting enter lots>

kurokobo commented 3 years ago

@nashford77 I've deployed new Ubuntu 20.04, K3s, AWX just now, everything works as expected 🤔 

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

$ kubectl -n awx get awx,all,ingress
NAME                      AGE
awx.awx.ansible.com/awx   5m9s

NAME                      READY   STATUS    RESTARTS   AGE
pod/awx-postgres-0        1/1     Running   0          4m53s
pod/awx-59ff55b5b-8hp99   4/4     Running   0          4m45s

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None         <none>        5432/TCP   4m53s
service/awx-service    ClusterIP   10.43.21.0   <none>        80/TCP     4m46s

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx   1/1     1            1           4m45s

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-59ff55b5b   1         1         1       4m45s

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     4m53s

NAME                                    CLASS    HOSTS             ADDRESS         PORTS     AGE
ingress.networking.k8s.io/awx-ingress   <none>   awx.example.com   192.168.0.218   80, 443   4m46s

curl also respond as expected.

$ curl 192.168.0.219
404 page not found

$ curl -k https://awx.example.com/
<!doctype html><html lang="en"><head><script nonce="xSOSPXwun2bwLidkIAIBMLegcEGQxs6CQ6X+gI5M71A=" typ...
nashford77 commented 3 years ago

Mine deploys fine, but it did not respond to a named call. I have wiped it many times, retried it etc. I have to be missing something. Did you not enable ip forwarding or anything special ?

On Thu, Aug 26, 2021, 11:30 AM kurokobo @.***> wrote:

@nashford77 https://github.com/nashford77 I've deployed new Ubuntu 20.04, K3s, AWX just now, everything works as expected 🤔

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 20.04.1 LTS

Release: 20.04

Codename: focal

$ kubectl -n awx get awx,all,ingress

NAME AGE awx.awx.ansible.com/awx 5m9s

NAME READY STATUS RESTARTS AGE

pod/awx-postgres-0 1/1 Running 0 4m53s

pod/awx-59ff55b5b-8hp99 4/4 Running 0 4m45s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/awx-postgres ClusterIP None 5432/TCP 4m53s

service/awx-service ClusterIP 10.43.21.0 80/TCP 4m46s

NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/awx 1/1 1 1 4m45s

NAME DESIRED CURRENT READY AGE

replicaset.apps/awx-59ff55b5b 1 1 1 4m45s

NAME READY AGE

statefulset.apps/awx-postgres 1/1 4m53s

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/awx-ingress awx.example.com 192.168.0.218 80, 443 4m46s

curl also respond as expected.

$ curl 192.168.0.219

404 page not found

$ curl -k https://awx.example.com/

<!doctype html><script nonce="xSOSPXwun2bwLidkIAIBMLegcEGQxs6CQ6X+gI5M71A=" typ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kurokobo/awx-on-k3s/issues/3#issuecomment-906515327, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFSSJUYZBO4XQ5FMWHQC7JTT6ZMZXANCNFSM5C2MTTNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

kurokobo commented 3 years ago

@nashford77 Try curl http://192.168.5.104/ -H "host: awx.tunninet.com"

kurokobo commented 3 years ago

Did you not enable ip forwarding or anything special

I haven't changed any OS settings except for /etc/hosts since the clean install of Ubuntu 20.04 Server. Current configuration around IP forwarding looks like this.

$ sudo sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0

$ sudo systemctl status ufw
● ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2021-08-26 15:16:21 UTC; 23min ago
       Docs: man:ufw(8)
   Main PID: 502 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 9448)
     Memory: 0B
     CGroup: /system.slice/ufw.service

$ grep awx /etc/hosts
192.168.0.218 awx.example.com
nashford77 commented 3 years ago

root@u500-cube-server:~/awx-on-k3s# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 21.04 Release: 21.04 Codename: hirsute

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx get awx,all,ingress NAME AGE awx.awx.ansible.com/awx 12h

NAME READY STATUS RESTARTS AGE pod/awx-postgres-0 1/1 Running 0 12h pod/awx-59ff55b5b-2czpx 4/4 Running 68 12h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-postgres ClusterIP None 5432/TCP 12h service/awx-service ClusterIP 10.43.251.54 80/TCP 12h

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx 1/1 1 1 12h

NAME DESIRED CURRENT READY AGE replicaset.apps/awx-59ff55b5b 1 1 1 12h

NAME READY AGE statefulset.apps/awx-postgres 1/1 12h

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/awx-ingress awx.tunninet.com 192.168.5.104 80, 443 12h

root@u500-cube-server:~/awx-on-k3s# curl 192.168.5.104 404 page not found

root@u500-cube-server:~/awx-on-k3s# ping awx.tunninet.com (ENABLED FOR TESTING ONLY) PING awx.tunninet.com (192.168.5.104) 56(84) bytes of data. 64 bytes from awx.tunninet.com (192.168.5.104): icmp_seq=1 ttl=64 time=0.121 ms 64 bytes from awx.tunninet.com (192.168.5.104): icmp_seq=2 ttl=64 time=0.083 ms

root@u500-cube-server:~/awx-on-k3s# curl -k https://awx.tunninet.com/

^C root@u500-cube-server:~/awx-on-k3s# curl http://192.168.5.104/ -H "host: awx.tunninet.com" ^C root@u500-cube-server:~/awx-on-k3s# sudo systemctl status ufw ● ufw.service - Uncomplicated firewall Loaded: loaded (/lib/systemd/system/ufw.service; disabled; vendor preset: enabled) Active: active (exited) since Thu 2021-08-26 15:41:16 UTC; 22s ago Docs: man:ufw(8) Process: 478512 ExecStart=/lib/ufw/ufw-init start quiet (code=exited, status=0/SUCCESS) Main PID: 478512 (code=exited, status=0/SUCCESS) root@u500-cube-server:~/awx-on-k3s# sudo sysctl -a | grep ip_forward net.ipv4.ip_forward = 1 net.ipv4.ip_forward_update_priority = 1 net.ipv4.ip_forward_use_pmtu = 0
nashford77 commented 3 years ago

i also tried w UFW down as well to test - no dice

nashford77 commented 3 years ago

oh - i am on 21.04 - i am guessing that's an issue :( Can you try on that code line ?

kurokobo commented 3 years ago

Now installing 21.04 on my lab 🚀

kurokobo commented 3 years ago

Wired, works good for me 🤔 

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 21.04
Release:        21.04
Codename:       hirsute

$ grep awx /etc/hosts
192.168.0.218 awx.tunninet.com

$ curl -k https://awx.tunninet.com/
<!doctype html><html lang="en"><head><script nonce="ZAShxq63O1EUgPzaBUxvuqDX4Cqdpb/91ow/0fIxbSg=" type="t...

Could you share the output of these commands? And just to confirm, is there any proxies?

k3s --version
kubectl -n kube-system get po
kubectl -n awx get ingress awx-ingress -o yaml
curl 192.168.5.104
curl http://192.168.5.104/
dig awx.tunninet.com
nashford77 commented 3 years ago

root@u500-cube-server:~/awx-on-k3s# k3s --version k3s version v1.21.3+k3s1 (1d1f220f) go version go1.16.6 root@u500-cube-server:~/awx-on-k3s# kubectl -n kube-system get po NAME READY STATUS RESTARTS AGE metrics-server-86cbb8457f-l9mnh 1/1 Running 0 13h local-path-provisioner-5ff76fc89d-v6tv2 1/1 Running 0 13h coredns-7448499f4d-kgnms 1/1 Running 0 13h helm-install-traefik-crd-wrhc7 0/1 Completed 0 13h helm-install-traefik-nbvwh 0/1 Completed 1 13h svclb-traefik-9s8vg 2/2 Running 0 13h traefik-97b44b794-n5j4t 1/1 Running 0 13h root@u500-cube-server:~/awx-on-k3s# kubectl -n awx get ingress awx-ingress -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"labels":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/name":"awx","app.kubernetes.io/operator-version":"0.13.0","app.kubernetes.io/part-of":"awx"},"name":"awx-ingress","namespace":"awx"},"spec":{"rules":[{"host":"awx.tunninet.com","http":{"paths":[{"backend":{"serviceName":"awx-service","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["awx.tunninet.com"],"secretName":"awx-secret-tls"}]}}' creationTimestamp: "2021-08-26T02:38:00Z" generation: 1 labels: app.kubernetes.io/component: awx app.kubernetes.io/managed-by: awx-operator app.kubernetes.io/name: awx app.kubernetes.io/operator-version: 0.13.0 app.kubernetes.io/part-of: awx name: awx-ingress namespace: awx ownerReferences:

; <<>> DiG 9.16.8-Ubuntu <<>> awx.tunninet.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52069 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;awx.tunninet.com. IN A

;; ANSWER SECTION: awx.tunninet.com. 0 IN A 192.168.5.104

;; Query time: 0 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Thu Aug 26 16:29:51 UTC 2021 ;; MSG SIZE rcvd: 61

``

nashford77 commented 3 years ago

can you show me a snap list as well ? wondering which snaps you choose or did not choose

kurokobo commented 3 years ago 
$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20210309  1997   latest/stable    canonical?  base
lxd     4.13      20037  latest/stable/…  canonical?  -
snapd   2.49.2    11588  latest/stable    canonical?  snapd
kurokobo commented 3 years ago

F.Y.I, it's good way to wrap your output with triple back-quotes.

$ sudo snap list Name Version Rev Tracking Publisher Notes core18 20210309 1997 latest/stable canonical? base lxd 4.13 20037 latest/stable/… canonical? - snapd 2.49.2 11588 latest/stable canonical? snapd

This will show your output as formatted strings.

kurokobo commented 3 years ago

I don't know if it will be helpful, can you try restarting treafik?

# Once deleted, new pods will be created and started again automatically
kubectl -n kube-system delete pod -l app.kubernetes.io/name=traefik
kubectl -n kube-system delete pod -l app=svclb-traefik

And just wait few minutes, check again; -v is the option to get verbose output.

curl -v http://192.168.5.104/
curl -v http://192.168.5.104/ -H "Host: example.com"
curl -v http://192.168.5.104/ -H "Host: awx.tunninet.com"
curl -v http://awx.tunninet.com/
nashford77 commented 3 years ago

root@u500-cube-server:~/awx-on-k3s# snap list Name Version Rev Tracking Publisher Notes core 16-2.51.4 11606 latest/stable canonical✓ core core18 20210722 2128 latest/stable canonical✓ base core20 20210702 1081 latest/stable canonical✓ base lxd 4.17 21260 latest/stable/… canonical✓ - snapd 2.51.4 12883 latest/stable canonical✓ snapd

nashford77 commented 3 years ago

deleted pods - waited 5-10....

root@u500-cube-server:~/awx-on-k3s# curl -v http://192.168.5.104/

^C root@u500-cube-server:~/awx-on-k3s# curl -v http://awx.tunninet.com/

still hangs but responds ? so odd

nashford77 commented 3 years ago

can you show the output of::

kubectl get svc --all-namespaces

kurokobo commented 3 years ago 
$ kubectl get svc --all-namespaces
NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default       kubernetes             ClusterIP      10.43.0.1       <none>          443/TCP                      99m
kube-system   kube-dns               ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       99m
kube-system   metrics-server         ClusterIP      10.43.132.143   <none>          443/TCP                      99m
default       awx-operator-metrics   ClusterIP      10.43.77.208    <none>          8383/TCP,8686/TCP            95m
kube-system   traefik                LoadBalancer   10.43.147.173   192.168.0.218   80:30107/TCP,443:31629/TCP   98m
awx           awx-postgres           ClusterIP      None            <none>          5432/TCP                     2m32s
awx           awx-service            ClusterIP      10.43.215.150   <none>          80/TCP                       2m26s
nashford77 commented 3 years ago

should the LB not point to the AWX-service ?

--> kubectl edit svc -n kube-system traefik <-- this shows me the below (i wanted to edit the ports as these are just how the system maps to the svc right ? can you should me this output ? should the IP not match up to AWX-SERVICE ?

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: kube-system creationTimestamp: "2021-08-26T02:34:45Z" labels: app.kubernetes.io/instance: traefik app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: traefik helm.sh/chart: traefik-9.18.2 name: traefik namespace: kube-system resourceVersion: "37977" uid: 5512d2e3-e339-4821-8de8-5236b73cba6d spec: clusterIP: 10.43.185.225 clusterIPs:

kurokobo commented 3 years ago

There is no problem if the treafik doesn't have a spec pointing to the AWX's IP address. Traefik is used commonly across the cluster and is not specific to AWX.

$ kubectl -n kube-system get svc traefik -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2021-08-26T15:59:15Z"
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.18.2
  name: traefik
  namespace: kube-system
  resourceVersion: "3124"
  uid: 51895f37-1076-4ac5-a854-4d9d3df13b12
spec:
  clusterIP: 10.43.147.173
  clusterIPs:
  - 10.43.147.173
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    nodePort: 30107
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 31629
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.0.218
kurokobo commented 3 years ago

Looking at the results of curl -v, it seems that Traefik is working, the requests was acceptted but no response...? Can you check the logs of awx-web container?

# Start watching
kubectl -n awx logs -f deployment.apps/awx -c awx-web

# Open another terminal and invoke curl
curl -v http://awx.tunninet.com/

Expected logs when accessing from curl is:

2021-08-26 17:48:52,271 DEBUG    [20dde60880e84f5da9907cb3414c2263] awx.analytics.performance request: <WSGIRequest: GET '/'>, response_time: 0.211s
10.42.0.16 - - [26/Aug/2021:17:48:52 +0000] "GET / HTTP/1.1" 200 1190 "-" "curl/7.74.0" "10.42.0.1"
[pid: 46|app: 0|req: 1/2] 10.42.0.16 () {46 vars in 582 bytes} [Thu Aug 26 17:48:52 2021] GET / => generated 1190 bytes in 212 msecs (HTTP/1.1 200) 7 headers in 269 bytes (1 switches on core 0)
kurokobo commented 3 years ago

Sorry, I'm out of time today. Please write it down and I'll look at it again later.

nashford77 commented 3 years ago

Nothing at all is logged - i just tried. So it would seem its talking to nothing somehow

nashford77 commented 3 years ago

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx logs -f deployment.apps/awx -c awx-web [wait-for-migrations] Waiting for database migrations... [wait-for-migrations] Attempt 1 of 30 [wait-for-migrations] Waiting 0.5 seconds before next attempt [wait-for-migrations] Attempt 2 of 30 [wait-for-migrations] Waiting 1 seconds before next attempt [wait-for-migrations] Attempt 3 of 30 [wait-for-migrations] Waiting 2 seconds before next attempt [wait-for-migrations] Attempt 4 of 30 [wait-for-migrations] Waiting 4 seconds before next attempt

kurokobo commented 3 years ago

oops, that logs mean simply your awx is not working...

nashford77 commented 3 years ago

i read that perhaps its stuck in a migrating state ? how would we manually force it ?

awx-manage migrate --noinput

how do i pass this to the container or log in to it with a shell of /bin/bash and do that ?

kurokobo commented 3 years ago

i'm out now so can't type exact command, but you can open shell by kubectl -n awx exec -it <pod name> -c <container name> -- bash. awx-manage is in awx-task container i think.

typically that log is caused by the connection issue between awx and postgres. i recommend wipe out everything by uninstalling k3s and manually delete data file under /data/postgres which used by PV and deploy again. The postgres credentials may be inconsistent with the data files, or it may be a problem with iptables or ufw.

nashford77 commented 3 years ago

No go ....

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx exec -it awx-59ff55b5b-2czpx -c awx-task --stdin --tty awx-task -- /bin/bash bash-4.4$ awx-manage migrate --noinput Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/init.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, kwasync) psycopg2.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/bin/awx-manage", line 8, in sys.exit(manage()) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/init.py", line 155, in manage if (connection.pg_version // 10000) < 12: File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/init.py", line 28, in getattr return getattr(connections[DEFAULT_DB_ALIAS], item) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 282, in pg_version with self.temporary_connection(): File "/usr/lib64/python3.8/contextlib.py", line 113, in enter return next(self.gen) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 593, in temporary_connection with self.cursor() as cursor: File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor return self._cursor() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor self.ensure_connection() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection self.connect() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect self.connection = self.get_new_connection(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection connection = Database.connect(conn_params) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/init.py", line 126, in connect conn = _connect(dsn, connection_factory=connection_factory, kwasync) django.db.utils.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known

bash-4.4$ command terminated with exit code 137 root@u500-cube-server:~/awx-on-k3s#

nashford77 commented 3 years ago

btw i also tried on aws-web - no go

nashford77 commented 3 years ago

how the heck to i delete this :) theres no deployment attached.... i think something stale is causing FW issues.

root@u500-cube-server:~# kubectl get all --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/helm-install-traefik-crd-r94hs 0/1 Pending 0 2m kube-system pod/helm-install-traefik-hdhq4 0/1 Pending 0 112s

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.43.0.1 443/TCP 31m

NAMESPACE NAME COMPLETIONS DURATION AGE kube-system job.batch/helm-install-traefik-crd 0/1 2m 2m kube-system job.batch/helm-install-traefik 0/1 112s 112s

nashford77 commented 3 years ago

they just return (cant set replica to 0 ... ) a delete comes right back

nashford77 commented 3 years ago

ok wow - success - it was a prior failed deployment from something other that k3s - anyways i cleaned up FW rules, the lot - took some hassle but its up ! thanks man!

kurokobo commented 3 years ago

Sorry I didn't get a chance to reply whole day. Thanks for letting me know it was resolved 😃