Orchestrate Multiple Selenoid Hubs

[ryanlevell]

I want to handle multiple selenoid hubs in a simple way. I am pretty new to both Docker and Kubernetes, but I want to run selenoid as a Kubernetes pod.

Currently, I was able to create the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: selenoid-pod
  labels:
    app: selenium-grid
spec:
  volumes:
    - name: test-cm-vol
      configMap:
        name: test-cm
    - name: docker-sock-vol
      hostPath:
        path: /var/run/docker.sock
  containers:
  - name: selenoid-hub
    image: aerokube/selenoid:latest
    env:
    - name: DOCKER_API_VERSION
      value: "1.24"
    resources:
      requests:
        cpu: 500m
        memory: 750Mi
    volumeMounts:
      - mountPath: /etc/selenoid/browsers.json
        name: test-cm-vol
        subPath: browsers.json
      - mountPath: /var/run/docker.sock
        name: docker-sock-vol

However, when I try to run a test I get:

Exception in thread "main" org.openqa.selenium.WebDriverException: create container: Error: No such image: selenoid/vnc:chrome_62.0 (WARNING: The server did not provide any stacktrace information)

This is because I do not have the docker images pre-pulled. This is where I am stuck. I am not sure how to pull the (newest) browser images from Kubernetes. This would also require an update to browsers.json right? Which is kind of the purpose to cm to handle these parts.

Or, is there some other way to handle multiple selenoid hubs simply? Or to prevent this, is there some flag I can use to force download of the browser images if they are missing?

[vania-pooh]

1) To distribute sessions across multiple Selenoids just use our load balancer - Ggr. 2) To pull images to Kubernetes you need to point your Docker client to every host. E.g. with Minikube this is done with: $ eval $(minikube docker-env). We are currently working on native Kubernetes Selenium solution called Moon. However it is commercial and closed-source. Selenoid is mainly expected to be run on virtual machines or hardware servers because when running inside Kubernetes it will launch browsers only on the same host where it runs. If you still wish to use Selenoid inside Kubernetes then browsers.json can be stored as ConfigMap and mounted to Selenoid pod as a file. 3) We intentionally do not implement auto-pull functionality. This guarantees cluster stability.

[ryanlevell]

I definitely plan to use Ggr. I just wanted my Selenoid hubs to be "always up" via a K8s Deployment that will restart the pod if necessary.

when running inside Kubernetes it will launch browsers only on the same host where it runs

Ok. I think that means I'll have to customize my own selenium grid via servlets/proxies then.

[aandryashin]

I recommend that you use daemonset for selenoid instances instead of deployment, in this case only one selenoid will run on every node. Also you nave to mount docker socket within pod to allow selenoid start containers on node. For ggr - deployment with service is ok.

2017-11-07 22:54 GMT+03:00 ryanlevell notifications@github.com:

I definitely plan to use Ggr. I just wanted my Selenoid hubs to be "always up" via a K8s Deployment that will restart the pod if necessary.

when running inside Kubernetes it will launch browsers only on the same host where it runs Ok. I think that means I'll have to customize my own selenium grid via servlets and proxies then.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aerokube/selenoid/issues/264#issuecomment-342602241, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWNba7S2nVk_oDdv0MENFYiMQz8nCWnks5s0LV0gaJpZM4QVORb .

[aandryashin]

@ryanlevell please take a look on our native Kubernetes implementation of Selenoid:

http://aerokube.com/moon/latest/

it has several significant advantages described here:

http://aerokube.com/moon/latest/#_moon_vs_selenoid

Alexander Andryashin

2017-11-08 12:37 GMT+03:00 Alexander Andryashin aandryashin@gmail.com:

I recommend that you use daemonset for selenoid instances instead of deployment, in this case only one selenoid will run on every node. Also you nave to mount docker socket within pod to allow selenoid start containers on node. For ggr - deployment with service is ok.

2017-11-07 22:54 GMT+03:00 ryanlevell notifications@github.com:

I definitely plan to use Ggr. I just wanted my Selenoid hubs to be "always up" via a K8s Deployment that will restart the pod if necessary.

when running inside Kubernetes it will launch browsers only on the same host where it runs Ok. I think that means I'll have to customize my own selenium grid via servlets and proxies then.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aerokube/selenoid/issues/264#issuecomment-342602241, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWNba7S2nVk_oDdv0MENFYiMQz8nCWnks5s0LV0gaJpZM4QVORb .

[ryanlevell]

Thanks @aandryashin I will take a look.

Uniform load distribution. Load balancers like Ggr provide are using random load distribution across available browser hosts. This makes them inefficient when overall load is above 80% of cluster capacity. Moon delivers exactly uniform distribution working good even under 100% load.

Moon does not use Ggr? Can you still route tests to cloud services in Moon?

[ryanlevell]

@vania-pooh

Selenoid is mainly expected to be run on virtual machines or hardware servers because when running inside Kubernetes it will launch browsers only on the same host where it runs

Selenoid also only launches browsers on the same host when run on a VM right? So the only difference would be a K8s pod vs a docker container. Or am I missing something?

[vania-pooh]

@ryanlevell the main difference between Kubernetes and Selenoid on Docker is that you get a cluster with unlimited size. Browser sessions are automatically distributed across all available hardware in Kubernetes quota. E.g. you can add more hardware (Kubernetes nodes) without touching Selenium configuration and then launch more browsers. It is even more fault-tolerant. We are showing the difference here. Our goal is to provide a rich UI for showing user sessions, logs, recorded videos and so on on top of Kubernetes. So in current state - we repeated Selenoid functionality on top of Kubernetes using native client library but plan to deliver more user-friendly features.

[vania-pooh]

@ryanlevell any more things to discuss here?