search

weaveworks / scope

Monitoring, visualisation & management for Docker & Kubernetes
https://www.weave.works/oss/scope/
Apache License 2.0
5.85k stars 709 forks source link

Can access scope only via port-forward but not NodePort or Ingress #3734

Closed TomaszKlosinski closed 4 years ago

TomaszKlosinski commented 4 years ago

Hi,

I've tried to expose scope via Traefik Ingress (and additionally HAProxy that we have in front of K8s cluster in my company) and it didn't worked. Initially I thought it's the problem with websockets support by Traefik-ingress and/or HAProxy. So I've decided to expose it via NodePort and access it directly. Still no success. My brower console is saying:

Firefox can’t establish a connection to the server at ws://k8s-master-01:32269/api/topology/hosts/ws?t=5s. app-4028151e529905aadd83.js:6:87430
The connection to ws://k8s-master-01:32269/api/topology/hosts/ws?t=5s was interrupted while the page was loading.

And in the pods logs I see no errors.

The only way that works is the port-froward example form the installation docs. Then it works like a charm. Anyone has a clue how to make it work via ingress?

Here's my manifest:

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: weave
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: weave-scope
      labels:
        name: weave-scope
      namespace: weave
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: weave-scope
      labels:
        name: weave-scope
    rules:
      - apiGroups:
          - ''
        resources:
          - pods
        verbs:
          - get
          - list
          - watch
          - delete
      - apiGroups:
          - ''
        resources:
          - pods/log
          - services
          - nodes
          - namespaces
          - persistentvolumes
          - persistentvolumeclaims
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - apps
        resources:
          - statefulsets
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - batch
        resources:
          - cronjobs
          - jobs
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - extensions
        resources:
          - deployments
          - daemonsets
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - extensions
        resources:
          - deployments/scale
        verbs:
          - get
          - update
      - apiGroups:
          - storage.k8s.io
        resources:
          - storageclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - volumesnapshot.external-storage.k8s.io
        resources:
          - volumesnapshots
          - volumesnapshotdatas
        verbs:
          - list
          - watch
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: weave-scope
      labels:
        name: weave-scope
    roleRef:
      kind: ClusterRole
      name: weave-scope
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: weave-scope
        namespace: weave
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: weave-scope-app
      labels:
        name: weave-scope-app
        app: weave-scope
        weave-cloud-component: scope
        weave-scope-component: app
      namespace: weave
    spec:
      replicas: 1
      revisionHistoryLimit: 2
      selector:
        matchLabels:
          name: weave-scope-app
          app: weave-scope
          weave-cloud-component: scope
          weave-scope-component: app
      template:
        metadata:
          labels:
            name: weave-scope-app
            app: weave-scope
            weave-cloud-component: scope
            weave-scope-component: app
        spec:
          containers:
            - name: app
              args:
                - '--weave=false'
                - '--mode=app'
              command:
                - /home/weave/scope
              env: []
              image: 'docker.io/weaveworks/scope:1.12.0'
              imagePullPolicy: IfNotPresent
              ports:
                - containerPort: 4040
                  protocol: TCP
  - apiVersion: v1
    kind: Service
    metadata:
      name: weave-scope-app
      labels:
        name: weave-scope-app
        app: weave-scope
        weave-cloud-component: scope
        weave-scope-component: app
      namespace: weave
    spec:
      type: NodePort  # Added by tkl
      ports:
        - name: app
          port: 4040
          protocol: TCP
          # targetPort: 30000
      selector:
        name: weave-scope-app
        app: weave-scope
        weave-cloud-component: scope
        weave-scope-component: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: weave-scope-cluster-agent
      labels:
        name: weave-scope-cluster-agent
        app: weave-scope
        weave-cloud-component: scope
        weave-scope-component: cluster-agent
      namespace: weave
    spec:
      replicas: 1
      revisionHistoryLimit: 2
      selector:
        matchLabels:
          name: weave-scope-cluster-agent
          app: weave-scope
          weave-cloud-component: scope
          weave-scope-component: cluster-agent
      template:
        metadata:
          labels:
            name: weave-scope-cluster-agent
            app: weave-scope
            weave-cloud-component: scope
            weave-scope-component: cluster-agent
        spec:
          containers:
            - name: scope-cluster-agent
              args:
                - '--weave=false'
                - '--mode=probe'
                - '--probe-only'
                - '--probe.kubernetes.role=cluster'
                - '--probe.http.listen=:4041'
                - '--probe.publish.interval=4500ms'
                - '--probe.spy.interval=2s'
                - 'weave-scope-app.weave.svc.cluster.local:4040'
              command:
                - /home/weave/scope
              env: []
              image: 'docker.io/weaveworks/scope:1.12.0'
              imagePullPolicy: IfNotPresent
              ports:
                - containerPort: 4041
                  protocol: TCP
              resources:
                requests:
                  cpu: 25m
                  memory: 80Mi
          serviceAccountName: weave-scope
  - apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: weave-scope-agent
      labels:
        name: weave-scope-agent
        app: weave-scope
        weave-cloud-component: scope
        weave-scope-component: agent
      namespace: weave
    spec:
      minReadySeconds: 5
      selector:
        matchLabels:
          name: weave-scope-agent
          app: weave-scope
          weave-cloud-component: scope
          weave-scope-component: agent
      template:
        metadata:
          labels:
            name: weave-scope-agent
            app: weave-scope
            weave-cloud-component: scope
            weave-scope-component: agent
        spec:
          containers:
            - name: scope-agent
              args:
                - '--weave=false'
                - '--probe.no-controls=true'
                - '--mode=probe'
                - '--probe-only'
                - '--probe.kubernetes.role=host'
                - '--probe.publish.interval=4500ms'
                - '--probe.spy.interval=2s'
                - '--probe.docker.bridge=docker0'
                - '--probe.docker=true'
                - 'weave-scope-app.weave.svc.cluster.local:4040'
              command:
                - /home/weave/scope
              env: []
              image: 'docker.io/weaveworks/scope:1.12.0'
              imagePullPolicy: IfNotPresent
              resources:
                requests:
                  cpu: 100m
                  memory: 100Mi
              securityContext:
                privileged: true
              volumeMounts:
                - name: scope-plugins
                  mountPath: /var/run/scope/plugins
                - name: sys-kernel-debug
                  mountPath: /sys/kernel/debug
                - name: docker-socket
                  mountPath: /var/run/docker.sock
          dnsPolicy: ClusterFirstWithHostNet
          hostNetwork: true
          hostPID: true
          tolerations:
            - effect: NoSchedule
              operator: Exists
          volumes:
            - name: scope-plugins
              hostPath:
                path: /var/run/scope/plugins
            - name: sys-kernel-debug
              hostPath:
                path: /sys/kernel/debug
            - name: docker-socket
              hostPath:
                path: /var/run/docker.sock
      updateStrategy:
        type: RollingUpdate
TomaszKlosinski commented 4 years ago

And here's some more info about the deployment:

⎈ kubectl describe deployment weave-scope-app -n weave
Name:                   weave-scope-app
Namespace:              weave
CreationTimestamp:      Tue, 10 Dec 2019 17:59:09 +0100
Labels:                 app=weave-scope
                        name=weave-scope-app
                        weave-cloud-component=scope
                        weave-scope-component=app
Annotations:            cloud.weave.works/launcher-info:
                          {
                            "original-request": {
                              "url": "/k8s/v1.10/scope.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNiIsIEdpdFZlcnNpb246InYxLjE2...
                              "date": "Tue Dec 10 2019 10:51:37 GMT+0000 (UTC)"
                            },
                            "email-address": "support@weave.works"
                          }
                        deployment.kubernetes.io/revision: 2
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"cloud.weave.works/launcher-info":"{\n  \"original-request\": {\n  ...
Selector:               app=weave-scope,name=weave-scope-app,weave-cloud-component=scope,weave-scope-component=app
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=weave-scope
           name=weave-scope-app
           weave-cloud-component=scope
           weave-scope-component=app
  Containers:
   app:
    Image:      docker.io/weaveworks/scope:1.12.0
    Port:       4040/TCP
    Host Port:  0/TCP
    Command:
      /home/weave/scope
    Args:
      --weave=false
      --mode=app
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   weave-scope-app-74b9997499 (1/1 replicas created)
Events:          <none>
⎈ kubectl describe svc weave-scope-app -n weave
Name:                     weave-scope-app
Namespace:                weave
Labels:                   app=weave-scope
                          name=weave-scope-app
                          weave-cloud-component=scope
                          weave-scope-component=app
Annotations:              cloud.weave.works/launcher-info:
                            {
                              "original-request": {
                                "url": "/k8s/v1.10/scope.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNiIsIEdpdFZlcnNpb246InYxLjE2...
                                "date": "Tue Dec 10 2019 10:51:37 GMT+0000 (UTC)"
                              },
                              "email-address": "support@weave.works"
                            }
                          kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.weave.works/launcher-info":"{\n  \"original-request\": {\n    \"url\...
Selector:                 app=weave-scope,name=weave-scope-app,weave-cloud-component=scope,weave-scope-component=app
Type:                     NodePort
IP:                       10.44.112.65
Port:                     app  4040/TCP
TargetPort:               4040/TCP
NodePort:                 app  32269/TCP
Endpoints:                10.45.4.20:4040
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
⎈ kubectl describe ds weave-scope-agent -n weave
Name:           weave-scope-agent
Selector:       app=weave-scope,name=weave-scope-agent,weave-cloud-component=scope,weave-scope-component=agent
Node-Selector:  <none>
Labels:         app=weave-scope
                name=weave-scope-agent
                weave-cloud-component=scope
                weave-scope-component=agent
Annotations:    cloud.weave.works/launcher-info:
                  {
                    "original-request": {
                      "url": "/k8s/v1.10/scope.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNiIsIEdpdFZlcnNpb246InYxLjE2...
                      "date": "Tue Dec 10 2019 10:51:37 GMT+0000 (UTC)"
                    },
                    "email-address": "support@weave.works"
                  }
                deprecated.daemonset.template.generation: 3
                kubectl.kubernetes.io/last-applied-configuration:
                  {"apiVersion":"apps/v1","kind":"DaemonSet","metadata":{"annotations":{"cloud.weave.works/launcher-info":"{\n  \"original-request\": {\n   ...
Desired Number of Nodes Scheduled: 6
Current Number of Nodes Scheduled: 6
Number of Nodes Scheduled with Up-to-date Pods: 6
Number of Nodes Scheduled with Available Pods: 6
Number of Nodes Misscheduled: 0
Pods Status:  6 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=weave-scope
           name=weave-scope-agent
           weave-cloud-component=scope
           weave-scope-component=agent
  Containers:
   scope-agent:
    Image:      docker.io/weaveworks/scope:1.12.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /home/weave/scope
    Args:
      --weave=false
      --mode=probe
      --probe-only
      --probe.kubernetes.role=host
      --probe.publish.interval=4500ms
      --probe.spy.interval=2s
      --probe.docker.bridge=docker0
      --probe.docker=true
      weave-scope-app.weave.svc.cluster.local:4040
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /sys/kernel/debug from sys-kernel-debug (rw)
      /var/run/docker.sock from docker-socket (rw)
      /var/run/scope/plugins from scope-plugins (rw)
  Volumes:
   scope-plugins:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/scope/plugins
    HostPathType:
   sys-kernel-debug:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/kernel/debug
    HostPathType:
   docker-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulDelete  54m   daemonset-controller  Deleted pod: weave-scope-agent-krzj4
  Normal  SuccessfulCreate  54m   daemonset-controller  Created pod: weave-scope-agent-dwhjl
  Normal  SuccessfulDelete  54m   daemonset-controller  Deleted pod: weave-scope-agent-2gnkl
  Normal  SuccessfulCreate  53m   daemonset-controller  Created pod: weave-scope-agent-6vbh4
  Normal  SuccessfulDelete  53m   daemonset-controller  Deleted pod: weave-scope-agent-jpkpl
TomaszKlosinski commented 4 years ago

Ok, I realised it works on Chrome, but not on Firefox. Issue can be closed.