connecting with a LB

[shlomitsur]

Hello Thanks for this great product. Since we've not using ggr here I tried to add a simple ingress. I dont think I did a good job since it's not working well. What is the preferred way to work with this cluster? Thanks

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: selenosis-ingress
  namespace: selenosis
  annotations:
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "60"
    alb.ingress.kubernetes.io/healthcheck-path: /wd/hub/status
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "50"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/ip-address-type: ipv4
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":4444}]'
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/security-groups: sg-06ae2e882fce41fb9
    alb.ingress.kubernetes.io/success-codes: 200-499
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60,load_balancing.algorithm.type=least_outstanding_requests,deregistration_delay.timeout_seconds=10
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "10"
    kubernetes.io/ingress.class: alb
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: selenosis
              servicePort: 4444

[shlomitsur]

Mabye the stickiness is causing issues because the seleniferous is doing the routing already?

[alcounit]

Hi @shlomitsur, thanks for the feedback. Can you please give more info about your cluster and what exactly is wrong when you are trying to use ingress?

[shlomitsur]

Hi @alcounit I run in EKS with ~ 100 replicas. I'm using selenium 3.3.1 I get a lot like this:

driver.get("https://google.com") Traceback (most recent call last): File "", line 1, in File "/home/ironscales/venv3/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "/home/ironscales/venv3/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/home/ironscales/venv3/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 208, in check_response raise exception_class(value) WebDriverException: Message:

[shlomitsur]

I also see that every session request is duplicated across all pods :) leading to "max session reached" error very fast

[alcounit]

Try to run your tests w/o your ingress, use service or nodeport Share your selenosis deployment manifest Logs of selenosis, will be great to have Output of kubectl get po -n selenosis after your tests started

[shlomitsur]

Thank you @alcounit I basically used your selenosis-deploy files and added one file 05-ingress.yaml: `apiVersion: extensions/v1beta1 kind: Ingress metadata: name: selenosis-ingress namespace: selenosis annotations: alb.ingress.kubernetes.io/healthcheck-interval-seconds: "60" alb.ingress.kubernetes.io/healthcheck-path: /wd/hub/status alb.ingress.kubernetes.io/healthcheck-port: traffic-port alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "50" alb.ingress.kubernetes.io/healthy-threshold-count: "2" alb.ingress.kubernetes.io/ip-address-type: ipv4 alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":4444}]' alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: sg-06ae2e882fce41fb9 alb.ingress.kubernetes.io/success-codes: 200-499 alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/unhealthy-threshold-count: "10" kubernetes.io/ingress.class: alb service.beta.kubernetes.io/aws-load-balancer-internal: "true" alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60,load_balancing.algorithm.type=least_outstanding_requests spec: rules:

• http: paths:
  • path: /* backend: serviceName: selenosis servicePort: 4444`

Unfortunately I just started a 10 days :) vacation from work so I can't show output. Will do that right when I'll get back to work. Thanks again @alcounit

[alcounit]

Sure, no problem, enjoy your vacation 👍

[shlomitsur]

I found the issue: The Cluster-Ip is not accessible in EKS because it needs to have a elb annotation: service.beta.kubernetes.io/aws-load-balancer-internal: "true" I ended up with this service configuration:

apiVersion: v1
kind: Service
metadata:
  name: selenosis
  namespace: selenosis
  annotations:
      alb.ingress.kubernetes.io/ip-address-type: ipv4
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 4444}]'
      alb.ingress.kubernetes.io/target-type: instance
      alb.ingress.kubernetes.io/scheme: internal
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/success-codes: 200-499
      alb.ingress.kubernetes.io/security-groups: sg-<your_sg_#>
      alb.ingress.kubernetes.io/healthcheck-port: traffic-port
      alb.ingress.kubernetes.io/healthcheck-path: /wd/hub/status
      alb.ingress.kubernetes.io/healthcheck-interval-seconds: '60'
      alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '50'
      alb.ingress.kubernetes.io/tags: Environment=prod,Team=test,Name=k8s-visual-inspector,app=k8s-visual-inspector
      alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60,load_balancing.algorithm.type=least_outstanding_requests,deregistration_delay.timeout_seconds=10
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: selenium
    port: 4444
    protocol: TCP
    targetPort: 4444
    nodePort: 31000
  selector:
    app: selenosis
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
  name: seleniferous
  namespace: selenosis
spec:
  selector:
    type: browser
  clusterIP: None
  publishNotReadyAddresses: true

[alcounit]

Good to hear, can I close the issue?