search

alcounit / selenosis

Scalable, stateless selenium hub for Kubernetes cluster
Apache License 2.0
81 stars 24 forks source link

Chrome Nodes get Stuck on "Waiting X server..." #32

Closed shlomitsur closed 3 years ago

shlomitsur commented 3 years ago

Hi, I'm using selenosis-delopy with image 'selenoid/chrome:89.0' After a while I see many 'selenoid-chrome-89-0-f2c...' containers stuck in 'Terminating' stage. The container log shows:

Waiting X server...
Waiting X server...
Waiting X server...
....

I'm not using VNC, the containers are stuck on waiting for X server. This is in the chrome entrypoint:

retcode=1
until [ $retcode -eq 0 ]; do
  DISPLAY="$DISPLAY" wmctrl -m >/dev/null 2>&1
  retcode=$?
  if [ $retcode -ne 0 ]; then
    echo Waiting X server...
    sleep 0.1
  fi
done

https://github.com/aerokube/images/blob/5d1e1fa807d7ad55029e87665c6d926b3085bd36/static/chrome/entrypoint.sh

How can I disable it? Thank you

alcounit commented 3 years ago

Hi @shlomitsur There is no need to disable X server, since it is needed for browser and I don't have any issue running selenoid/chrome:89.0 on my cluster

/home/alcounit/stern -n selenosis --selector='type=browser'
+ selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 › seleniferous
+ selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 › browser
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser 2021/04/11 13:21:05 [INIT] [Listening on :7070]
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser 20
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Logging to: /dev/null
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"starting seleniferous v0.0.3-develop","time":"2021-04-11T13:21:05Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"kubernetes client created","time":"2021-04-11T13:21:05Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Waiting X server...
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Starting ChromeDriver 89.0.4389.23 (61b08ee2c50024bab004e48d2b1b083cdbdac579-refs/branch-heads/4389@{#294}) on port 4444
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser All remote connections are allowed. Use an allowlist instead!
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 browser ChromeDriver was started successfully.
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"new session request","request":"POST /wd/hub/session","request_by":"selenosis-76479d6b89-cmwfp","request_id":"8e6d5060-8790-40bc-9176-2df53a5cf226","time":"2021-04-11T13:21:06Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"new session request completed: 8b6500c8ab4d93192788032caa289f86","request":"POST /wd/hub/session","request_by":"selenosis-76479d6b89-cmwfp","request_id":"8e6d5060-8790-40bc-9176-2df53a5cf226","time":"2021-04-11T13:21:07Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"GET /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/window_handle","request_by":"selenosis-76479d6b89-cmwfp","request_id":"02c13762-8602-4f26-8cfb-e040f29b4a0d","time":"2021-04-11T13:21:07Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"GET /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/window_handle","request_by":"selenosis-76479d6b89-cmwfp","request_id":"95e595e1-81b8-4c48-8a76-5e46728a940b","time":"2021-04-11T13:21:07Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"POST /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/window/CDwindow-1FB7CCF49457FFC6A940327EA772985A/maximize","request_by":"selenosis-76479d6b89-cmwfp","request_id":"92c70bb1-a969-4f63-9f50-c1687df32c48","time":"2021-04-11T13:21:08Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"POST /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/url","request_by":"selenosis-76479d6b89-cmwfp","request_id":"739fc354-1a14-4b08-a032-e49a62bffb8d","time":"2021-04-11T13:21:08Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"GET /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/screenshot","request_by":"selenosis-76479d6b89-cmwfp","request_id":"b84d60df-6a6b-42f2-a82f-b0adf7dfab68","time":"2021-04-11T13:21:11Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"POST /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/element","request_by":"selenosis-76479d6b89-cmwfp","request_id":"cf9657eb-d61e-4afb-8e40-c6ef211cb1bd","time":"2021-04-11T13:21:43Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"POST /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/element/0.06913011458384544-1/click","request_by":"selenosis-76479d6b89-cmwfp","request_id":"cec9bf48-2419-4177-97ef-50f3f61432b0","time":"2021-04-11T13:21:43Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"GET /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9/screenshot","request_by":"selenosis-76479d6b89-cmwfp","request_id":"491c669e-8c43-4b90-b58e-7a3c2a8c407d","time":"2021-04-11T13:21:46Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"warning","msg":"session selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 delete request","request":"DELETE /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9","request_by":"selenosis-76479d6b89-cmwfp","request_id":"aa610170-a51d-4a59-8e99-0c106e9935ce","time":"2021-04-11T13:21:51Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"info","msg":"proxy session","request":"DELETE /wd/hub/session/selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9","request_by":"selenosis-76479d6b89-cmwfp","request_id":"aa610170-a51d-4a59-8e99-0c106e9935ce","time":"2021-04-11T13:21:51Z"}
selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9 seleniferous {"level":"warning","msg":"stopping seleniferous","time":"2021-04-11T13:21:51Z"}
- selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9
- selenoid-chrome-89-0-96353e2f-1c47-4007-a6e4-21344623a2a9

Share your browsers config, selenosis logs and logs from browser pod(for both containers)

shlomitsur commented 3 years ago

browsers.yaml:

chrome:
  defaultVersion: "89.0"
  path: "/"
  kernelCaps:
  - SYS_ADMIN
  versions:
    '89.0':
        image: selenoid/chrome:89.0

deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenosis
  namespace: selenosis
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  replicas: 2
  selector:
    matchLabels:
      app: selenosis
  template:
    metadata:
      labels:
        app: selenosis
        selenosis.app.type: worker
      namespace: selenosis
    spec:
      containers:
      - args: ["/selenosis", "--browsers-config", "./config/browsers.yaml",
               "--namespace", "selenosis", "--service-name", "seleniferous",
               "--browser-limit", "500  ", "--proxy-image",
               "alcounit/seleniferous:v0.0.3-develop",
               "--browser-wait-timeout", "1m10s", "--session-wait-timeout",
               "0m30s", "--session-idle-timeout", "0m30s",
               "--session-retry-count", "10"]
        image: alcounit/selenosis:v1.0.2
        name: selenosis
        ports:
        - containerPort: 4444
          name: selenium
          protocol: TCP
        volumeMounts:
        - mountPath: ./config
          name: browsers-config
        imagePullPolicy: IfNotPresent
        readinessProbe:
          httpGet:
            path: /healthz
            port: 4444
          periodSeconds: 2
          initialDelaySeconds: 3
        livenessProbe:
          httpGet:
            path: /healthz
            port: 4444
          periodSeconds: 2
          initialDelaySeconds: 3
        resources:
          requests:
            memory: 500Mi
            cpu: 1500m
      volumes:
      - name: browsers-config
        configMap:
          name: selenosis-config
$ kubectl describe pod selenoid-chrome-89-0-4c6328f8-7ec5-4227-b3c1-533e9cd6dfbe -n selenosis
Name:                      selenoid-chrome-89-0-4c6328f8-7ec5-4227-b3c1-533e9cd6dfbe
Namespace:                 selenosis
Priority:                  0
Node:                      ip-10-10-20-246.us-west-2.compute.internal/10.10.20.246
Start Time:                Sun, 11 Apr 2021 13:38:27 +0300
Labels:                    selenosis.app.type=browser
                           session=selenoid-chrome-89-0-4c6328f8-7ec5-4227-b3c1-533e9cd6dfbe
                           type=browser
Annotations:               capabilities: {"browserName":"chrome","browserVersion":"89.0","testName":""}
                           kubernetes.io/psp: eks.privileged
Status:                    Terminating (lasts 3h51m)
Termination Grace Period:  15s
IP:                        10.10.30.163
IPs:
  IP:  10.10.30.163
Containers:
  browser:
    Container ID:   docker://875273ace4ef41105bb73d8bb4785ac86b701b161f738ba4b9f2ec5c420756f0
    Image:          selenoid/chrome:89.0
    Image ID:       docker-pullable://selenoid/chrome@sha256:0d205b40d563e4c851a90baf8e2bcc44a5dd691f30c3716e7da0741ef5f412dd
    Ports:          5900/TCP, 4444/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 11 Apr 2021 13:44:36 +0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /dev/shm from dshm (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tcqrc (ro)
  seleniferous:
    Container ID:  docker://9be87b04bd5e46bd55d9c095eeb573c6db087e5c69414ba55e8c03162e50e277
    Image:         alcounit/seleniferous:v0.0.3-develop
    Image ID:      docker-pullable://alcounit/seleniferous@sha256:7ef53bd97dd33cb0c79af7815acbfdc74217592c54c34cdb7f6cff87aa2ac5f3
    Port:          4445/TCP
    Host Port:     0/TCP
    Command:
      /seleniferous
      --listhen-port
      4445
      --proxy-default-path
      /session
      --idle-timeout
      30s
      --namespace
      selenosis
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tcqrc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  default-token-tcqrc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tcqrc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason         Age                     From     Message
  ----     ------         ----                    ----     -------
  Warning  FailedKillPod  2m34s (x91 over 3h45m)  kubelet  error killing pod: failed to "KillContainer" for "browser" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"

$ kubectl logs selenoid-chrome-89-0-4c6328f8-7ec5-4227-b3c1-533e9cd6dfbe -n selenosis browser
2021/04/11 10:44:36 [INIT] [Listening on :7070]
20
Waiting X server...
Logging to: /dev/null
Waiting X server...
Waiting X server...
Waiting X server...
Waiting X server...
Waiting X server...
Waiting X server...
Waiting X server...
....

Thank you @alcounit

alcounit commented 3 years ago

@shlomitsur hm... it is strange, issue reproduces only with chrome 89? What about other browsers? Please share logs from seleniferous sidecar container and selenoid, this will be really helpful

kubectl logs selenoid-chrome-89-0-ec893d4e-e2bb-485f-8372-a6b7e3413ffb -n selenosis seleniferous
shlomitsur commented 3 years ago

I'm closing this issue because I think it was my mistake - It was solved once I reduced the node instance size (I think there's a limit to # of xvfb connections per instance)