search

vmware-archive / kubeless

Kubernetes Native Serverless Framework
https://kubeless.io
Apache License 2.0
6.86k stars 754 forks source link

kubeless creates ingress object without address #1000

Open bilalrazamoon opened 5 years ago

bilalrazamoon commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report

What happened: kubeless creates ingress object without address

What you expected to happen: kubeless creates ingress object with address

How to reproduce it (as minimally and precisely as possible): I deploy AWS ALB (Application Load Balancer) with the help of this guide: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/setup/#kubectl

I deployed function:

NAME    NAMESPACE   HANDLER         RUNTIME     DEPENDENCIES    STATUS   
echo    default     echo.handler    python2.7                   1/1 READY

and created http trigger:

NAME    NAMESPACE   FUNCTION NAME
echo    default     echo

it created ingress object without address:

NAME   HOSTS             ADDRESS   PORTS   AGE
echo   foo.example.com             80      11m

Anything else we need to know?:

Environment:

andresmgot commented 5 years ago

hi @bilalrazamoon,

the issue is probably with the ingress provider. Can you see anything useful in the ingress controller logs? You can also get a bit more of information if you get the ingress as a YAML or if you describe it. From the documentation you point it seems that you can get the logs of the controller executing:

kubectl logs -n kube-system $(kubectl get po -n kube-system | egrep -o alb-ingress[a-zA-Z0-9-]+)

Also you can get the information about the ingress object executing:

kubectl get ingress -o  yaml echo
kubectl describe ingress echo
bilalrazamoon commented 5 years ago

@andresmgot Thanks for reply, here are

ingress controller logs:

-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.1.0
  Build:      git-72962fcb
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

W0211 21:19:03.609534       1 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0211 21:19:03.758939       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0211 21:19:03.759372       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0211 21:19:03.759435       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0211 21:19:03.759686       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0211 21:19:03.760423       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0211 21:19:03.760626       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0211 21:19:03.760930       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I0211 21:19:03.761386       1 leaderelection.go:187] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I0211 21:19:21.949840       1 leaderelection.go:196] successfully acquired lease kube-system/ingress-controller-leader-alb
I0211 21:19:22.050291       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I0211 21:19:22.150477       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1

ingress as a YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: "2019-02-11T21:22:26Z"
  generation: 1
  labels:
    created-by: kubeless
  name: echo
  namespace: default
  ownerReferences:
  - apiVersion: kubeless.io/v1beta1
    kind: HTTPTrigger
    name: echo
    uid: 21673fae-2e43-11e9-b35a-02e5e70866a0
  resourceVersion: "14506"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/echo
  uid: 216b1ec7-2e43-11e9-b35a-02e5e70866a0
spec:
  rules:
  - host: foo.example.com
    http:
      paths:
      - backend:
          serviceName: echo
          servicePort: 8080
        path: /
status:
  loadBalancer: {}

ingress describe:

Name:             echo
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  foo.example.com  
                   /   echo:8080 (<none>)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:                                        <none>
andresmgot commented 5 years ago

As far as I can see in the logs, the controller is not detecting the ingress object. Does it work for you if you follow this example?

https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/walkthrough/echoserver/

You may need to add some annotations to the ingress in order for the controller to detect it.

bilalrazamoon commented 5 years ago

Yes, it created ingress object with address:

NAME         HOSTS                    ADDRESS                                                                 PORTS   AGE
echoserver   echoserver.example.com   b0ef81a7-echoserver-echose-2ad7-710952959.us-east-1.elb.amazonaws.com   80      34s

ingress controller logs:

E0212 10:50:07.738191       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="no object matching key \"echoserver/echoserver\" in local store"  "controller"="alb-ingress-controller" "request"={"Namespace":"echoserver","Name":"echoserver"}
I0212 10:50:08.794372       1 loadbalancer.go:185] echoserver/echoserver: creating LoadBalancer b0ef81a7-echoserver-echose-2ad7
I0212 10:50:09.551836       1 loadbalancer.go:201] echoserver/echoserver: LoadBalancer b0ef81a7-echoserver-echose-2ad7 created, ARN: arn:aws:elasticloadbalancing:us-east-1:256578267670:loadbalancer/app/b0ef81a7-echoserver-echose-2ad7/ff6503a6c996e975
I0212 10:50:09.672013       1 targets.go:73] echoserver/echoserver: Adding targets to arn:aws:elasticloadbalancing:us-east-1:256578267670:targetgroup/b0ef81a7-248bd81323b6fa1c186/05d39c433751bee9: 172.31.64.15:8080
I0212 10:50:09.878529       1 targets.go:88] echoserver/echoserver: Removing targets from arn:aws:elasticloadbalancing:us-east-1:256578267670:targetgroup/b0ef81a7-248bd81323b6fa1c186/05d39c433751bee9: 172.31.60.234:8080
I0212 10:50:09.916794       1 listener.go:83] echoserver/echoserver: creating listener 80
I0212 10:50:09.969428       1 rules.go:60] echoserver/echoserver: creating rule 1 on arn:aws:elasticloadbalancing:us-east-1:256578267670:listener/app/b0ef81a7-echoserver-echose-2ad7/ff6503a6c996e975/80d1bedbf158cb5f
I0212 10:50:09.992964       1 rules.go:77] echoserver/echoserver: rule 1 created with conditions [{    Field: "host-header",    Values: ["echoserver.example.com"]  },{    Field: "path-pattern",    Values: ["/"]  }]
I0212 10:50:10.163617       1 lb_attachment.go:30] echoserver/echoserver: modify securityGroup on LoadBalancer arn:aws:elasticloadbalancing:us-east-1:256578267670:loadbalancer/app/b0ef81a7-echoserver-echose-2ad7/ff6503a6c996e975 to be [sg-0d1f2d7ac8bcbf2c2]
I0212 10:50:10.642618       1 instance_attachment.go:110] echoserver/echoserver: detaching securityGroup sg-00bf9b2f4a9c95473 from ENI eni-076b520b818f968d0
I0212 10:50:10.879363       1 instance_attachment.go:87] echoserver/echoserver: attaching securityGroup sg-00bf9b2f4a9c95473 to ENI eni-00cc725e43f4d14ea

ingress as a YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-412db124,subnet-adab5e86
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
    alb.ingress.kubernetes.io/target-type: ip
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/subnets":"subnet-412db124,subnet-adab5e86","alb.ingress.kubernetes.io/tags":"Environment=dev,Team=test","alb.ingress.kubernetes.io/target-type":"ip","kubernetes.io/ingress.class":"alb"},"name":"echoserver","namespace":"echoserver"},"spec":{"rules":[{"host":"echoserver.example.com","http":{"paths":[{"backend":{"serviceName":"echoserver","servicePort":80},"path":"/"}]}}]}}
    kubernetes.io/ingress.class: alb
  creationTimestamp: "2019-02-12T10:50:07Z"
  generation: 1
  name: echoserver
  namespace: echoserver
  resourceVersion: "111273"
  selfLink: /apis/extensions/v1beta1/namespaces/echoserver/ingresses/echoserver
  uid: f645c0fb-2eb3-11e9-b35a-02e5e70866a0
spec:
  rules:
  - host: echoserver.example.com
    http:
      paths:
      - backend:
          serviceName: echoserver
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - hostname: b0ef81a7-echoserver-echose-2ad7-710952959.us-east-1.elb.amazonaws.com

ingress describe:

Name:             echoserver
Namespace:        echoserver
Address:          b0ef81a7-echoserver-echose-2ad7-710952959.us-east-1.elb.amazonaws.com
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  echoserver.example.com  
                          /   echoserver:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/subnets":"subnet-412db124,subnet-adab5e86","alb.ingress.kubernetes.io/tags":"Environment=dev,Team=test","alb.ingress.kubernetes.io/target-type":"ip","kubernetes.io/ingress.class":"alb"},"name":"echoserver","namespace":"echoserver"},"spec":{"rules":[{"host":"echoserver.example.com","http":{"paths":[{"backend":{"serviceName":"echoserver","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:            alb
  alb.ingress.kubernetes.io/scheme:       internet-facing
  alb.ingress.kubernetes.io/subnets:      subnet-412db124,subnet-adab5e86
  alb.ingress.kubernetes.io/tags:         Environment=dev,Team=test
  alb.ingress.kubernetes.io/target-type:  ip
Events:
  Type    Reason  Age    From                    Message
  ----    ------  ----   ----                    -------
  Normal  CREATE  4m30s  alb-ingress-controller  LoadBalancer b0ef81a7-echoserver-echose-2ad7 created, ARN: arn:aws:elasticloadbalancing:us-east-1:256578267670:loadbalancer/app/b0ef81a7-echoserver-echose-2ad7/ff6503a6c996e975
  Normal  CREATE  4m30s  alb-ingress-controller  rule 1 created with conditions [{    Field: "host-header",    Values: ["echoserver.example.com"]  },{    Field: "path-pattern",    Values: ["/"]  }]
andresmgot commented 5 years ago

So the issue is that you need at least some of the annotations that you see in that example for the controller to pick it:

    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb

If you edit the ingress object generated by kubeless (kubectl edit ingress echo) you can copy those annotations there and you should be able to obtain a public address.

To support that in Kubeless out of the box we would need to extend the HTTP Trigger Controller to support alb ingresses the same way we do with Nginx or traefik. You can see some related code here:

https://github.com/kubeless/http-trigger/blob/master/pkg/utils/k8sutil.go#L270

Right now I don't have the resources to focus on this but if you want to send a PR for adding support for that I will be more than happy to help with that.