search

kubecube-io / KubeCube

KubeCube is an open source enterprise-level container platform
https://kubecube.io
Apache License 2.0
506 stars 70 forks source link

[Bug]日志查询报错，开启logersee后 request elasticsearch fail #126

Open snpyeso opened 2 years ago

snpyeso commented 2 years ago

使用日志采集

apiVersion: netease.com/v1
kind: Logconfig
metadata:
  creationTimestamp: '2022-06-13T05:59:27Z'
  generation: 1
  labels:
    app: dep-staff
  managedFields:
    -
      apiVersion: netease.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app': {}
        'f:spec':
          .: {}
          'f:inputs': {}
      manager: Mozilla
      operation: Update
      time: '2022-06-13T05:59:27Z'
  name: stufflog
  namespace: cqdx
  resourceVersion: '50486'
  uid: 4c2ba645-762e-4354-bae4-d9cddd1e18b4
spec:
  inputs:
    -
      enable: true
      type:
        name: dockerStdout

image

image 点击日志查询 image 报错的接口为 image

检查热插拔组件

[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME            PHASE   AGE
common          fail    2d2h
pivot-cluster   fail    2d2h

查看组件

spec:
  component:
    -
      name: audit
      status: disabled
    -
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: disabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: disabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://10.10.10.44:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: disabled
status:
  phase: fail
  results:
    -
      message: 'audit is disabled'
      name: audit
      result: success
      status: disabled
    -
      message: uninstalled
      name: logseer
      result: success
      status: disabled
    -
      message: 'release is running'
      name: logagent
      result: success
      status: enabled
    -
      message: 'release is running'
      name: elasticsearch
      result: success
      status: enabled
    -
      message: 'helm install fail, cannot re-use a name that is still in use'
      name: kubecube-monitoring
      result: fail
      status: enabled
    -
      message: 'release is running'
      name: kubecube-thanos
      result: success
      status: enabled

看了下官方文档也没说如何才能开启日志查询

tiancandevloper commented 2 years ago

https://www.kubecube.io/docs/user-guide/administration/audit/ 可以参考这篇文章，将logseer和logagent开启

zhujf1989 commented 2 years ago

可以参考文档：https://www.kubecube.io/docs/installation-guide/enable-plugins/ 修改HotPlug的配置, 将logseer及logagent enable

- name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: enabled                            ## 启停标识：这里disabled为禁用
- name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识：这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换
snpyeso commented 2 years ago

可以参考文档：https://www.kubecube.io/docs/installation-guide/enable-plugins/ 修改HotPlug的配置, 将logseer及logagent enable

- name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: enabled                            ## 启停标识：这里disabled为禁用
- name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识：这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换

这篇文章我看过，但是我的hotplug状态为fail 不知道有没有影响，文档中是runing,看了一会儿也没发现如何让它变成running状态

[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME            PHASE   AGE
common          fail    2d2h
pivot-cluster   fail    2d2h

将logseer及logagent enable 之后


kubectl get pod --all-namespaces
NAMESPACE             NAME                                                      READY   STATUS      RESTARTS       AGE
cqdxdevspace          dep-prepare-6596d79cf-mcmqw                               1/1     Running     0              19h
elasticsearch         elasticsearch-master-0                                    1/1     Running     0              2d
hnc-system            hnc-controller-manager-6d47799b76-kp4wl                   1/1     Running     1 (2d ago)     2d
ingress-nginx         ingress-nginx-admission-create-ggrhg                      0/1     Completed   0              2d
ingress-nginx         ingress-nginx-admission-patch-chxhk                       0/1     Completed   1              2d
ingress-nginx         ingress-nginx-controller-776d88d5c9-mthdn                 1/1     Running     0              2d
kube-system           calico-kube-controllers-d84f7b88b-krrjs                   1/1     Running     0              2d
kube-system           calico-node-b55ds                                         1/1     Running     0              2d
kube-system           coredns-65c54cc984-l4tpt                                  1/1     Running     0              2d
kube-system           coredns-65c54cc984-rq6b8                                  1/1     Running     0              2d
kube-system           etcd-localhost.localdomain                                1/1     Running     8              2d
kube-system           kube-apiserver-localhost.localdomain                      1/1     Running     0              26h
kube-system           kube-controller-manager-localhost.localdomain             1/1     Running     28 (26h ago)   2d
kube-system           kube-proxy-7bgj5                                          1/1     Running     0              2d
kube-system           kube-scheduler-localhost.localdomain                      1/1     Running     22 (26h ago)   2d
kube-system           metrics-server-6d69ff96c6-7s27b                           1/1     Running     3 (26h ago)    2d
kubecube-monitoring   kubecube-monitoring-kube-state-metrics-6cf5c8c6d4-dqv2t   1/1     Running     0              2d
kubecube-monitoring   kubecube-monitoring-operator-5f84949849-sjtwq             1/1     Running     0              2d
kubecube-monitoring   kubecube-monitoring-prometheus-node-exporter-c9cdt        1/1     Running     0              2d
kubecube-monitoring   kubecube-thanos-query-86f669566f-ndwm7                    1/1     Running     0              2d
kubecube-monitoring   kubecube-thanos-receive-0                                 1/1     Running     0              2d
kubecube-monitoring   prometheus-kubecube-monitoring-prometheus-0               2/2     Running     0              2d
kubecube-system       audit-78b7d4cf7d-4h2hq                                    1/1     Running     0              2d
kubecube-system       cloud-shell-5c87948759-4zcwq                              1/1     Running     0              2d
kubecube-system       frontend-7cbccf84fb-vh5hv                                 1/1     Running     0              2d
kubecube-system       kubecube-78b59d768d-tpssk                                 1/1     Running     3 (26h ago)    2d
kubecube-system       warden-d876d9fd8-969q5                                    1/1     Running     0              2d
kubecube-system       webconsole-7d7f989996-wvh2r                               1/1     Running     3 (26h ago)    2d
local-path-storage    local-path-provisioner-86b7fd94b5-gh2wl                   1/1     Running     3 (26h ago)    2d
logagent              filebeat-dhwr5                                            2/2     Running     0              6m16s
没有发现 logseer相关的pod

查看common yaml 发现logseer 为uninstalled
```yaml
spec:
  component:
    -
      name: audit
      status: disabled
    -
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: enabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: enabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://10.10.10.44:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: disabled
status:
  phase: fail
  results:
    -
      message: 'audit is disabled'
      name: audit
      result: success
      status: disabled
    -
      message: uninstalled
      name: logseer
      result: success
      status: disabled
    -
      message: 'release is running'
      name: logagent
      result: success
      status: enabled
    -
      message: 'release is running'
      name: elasticsearch
      result: success
      status: enabled
    -
      message: 'helm install fail, cannot re-use a name that is still in use'
      name: kubecube-monitoring
      result: fail
      status: enabled
    -
      message: 'release is running'
      name: kubecube-thanos
      result: success
      status: enabled

检查下发现 有common和pivot-cluster 的热插拔，发现common 和 pivot-cluster都有 logseer配置，小白都先开启吧，然后再开启elasticsearch, 呃…… 这次错误提示还是蛮明显的，但是不知道如何修复 {"reason":400,"code":400,"message":"request elasticsearch fail"} image 检查下elasticsearch的服务 发现一切正常

[root@localhost ~]# kubectl get all -n elasticsearch
NAME                         READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0   1/1     Running   0          147m

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch-master            ClusterIP   10.108.228.229   <none>        9200/TCP,9300/TCP   147m
service/elasticsearch-master-headless   ClusterIP   None             <none>        9200/TCP,9300/TCP   147m

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   1/1     147m
zhujf1989 commented 2 years ago

/cc @weilaaa

snpyeso commented 2 years ago

重新安装kubecube，使用的最新版，全部开启common里的插件和pivot-cluster里的插件，进入日志查询,依然 request elasticsearch fail 安装结果也和上面差不多

snpyeso commented 2 years ago

@fusidamj @hzliangbin @tiancandevloper @zhujf1989

ethfoo commented 2 years ago

麻烦发下请求的时候logseer的日志

snpyeso commented 2 years ago

麻烦发下请求的时候logseer的日志

好的稍等，我得重装一下，因为我尝试自己修改已经有点改得记不到哪里要恢复了

snpyeso commented 2 years ago

@ethfoo 安装了两台机器

[root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME                              STATUS   ROLES                  AGE    VERSION
zpfrltgup4tujpi1-0001.novalocal   Ready    control-plane,master   7m3s   v1.23.5
zpfrltgup4tujpi1.novalocal        Ready    node                   32s    v1.23.5

进入空间配置好基本环境 新增租户空间等

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME                         STATUS   AGE
default                      Active   13m
elasticsearch                Active   12m
hnc-system                   Active   12m
ingress-nginx                Active   12m
kube-node-lease              Active   13m
kube-public                  Active   13m
kube-system                  Active   13m
kubecube-monitoring          Active   13m
kubecube-project-wordpress   Active   113s
kubecube-system              Active   12m
kubecube-tenant-tenant       Active   2m2s
local-path-storage           Active   12m
wordpresspace                Active   50s

添加一个nginx 测试deployment

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n  wordpresspace
NAME                          READY   STATUS    RESTARTS   AGE
pod/dep-ng-5f6b6f4778-j5vb2   1/1     Running   0          62s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dep-ng   1/1     1            1           62s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/dep-ng-5f6b6f4778   1         1         1       62s

开启全部热插件

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name:         common
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:13Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:01Z
  Resource Version:  6058
  UID:               fc94d067-3712-4907-aca6-48d7f4095314
Spec:
  Component:
    Name:       audit
    Status:     enabled
    Name:       logseer
    Namespace:  logseer
    Pkg Name:   logseer-v1.0.0.tgz
    Status:     enabled
    Env:        clustername: "{{.cluster}}"

    Name:       logagent
    Namespace:  logagent
    Pkg Name:   logagent-v1.0.0.tgz
    Status:     enabled
    Name:       elasticsearch
    Namespace:  elasticsearch
    Pkg Name:   elasticsearch-7.8.1.tgz
    Status:     enabled
    Env:        grafana:
  enabled: false
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://172.31.0.171:31291/api/v1/receive

    Name:       kubecube-monitoring
    Namespace:  kubecube-monitoring
    Pkg Name:   kubecube-monitoring-15.4.12.tgz
    Status:     enabled
    Name:       kubecube-thanos
    Namespace:  kubecube-monitoring
    Pkg Name:   thanos-3.18.0.tgz
    Status:     enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled
Events:       <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name:         pivot-cluster
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:15Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:09Z
  Resource Version:  6066
  UID:               098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
  Component:
    Name:    logseer
    Status:  enabled
    Env:     grafana:
  enabled: true
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://kubecube-thanos-receive:19291/api/v1/receive

    Name:  kubecube-monitoring
    Env:   receive:
  tsdbRetention: 7d
  replicaCount: 1
  replicationFactor: 1

    Name:    kubecube-thanos
    Status:  enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled

配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng

image

点击日志查询 image

查看logseer日志

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer 
NAME                           READY   STATUS    RESTARTS   AGE
pod/logseer-657645d857-w5rv5   1/1     Running   0          6m11s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/logseer   ClusterIP   10.105.118.70   <none>        80/TCP    6m11s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/logseer   1/1     1            1           6m11s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/logseer-657645d857   1         1         1       6m11s

##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpresspace"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "*"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1657675115512,
                "lte" : 1657678715512,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1d",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-07-13 10:22:12.384 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
    at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
    at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
    at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
    at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
    at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
    at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
    at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
    at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
    at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
    at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
snpyeso commented 2 years ago

@ethfoo @zhujf1989 @tiancandevloper @hzliangbin 可以看一看么

snpyeso commented 2 years ago

@ethfoo 安装了两台机器

[root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME                              STATUS   ROLES                  AGE    VERSION
zpfrltgup4tujpi1-0001.novalocal   Ready    control-plane,master   7m3s   v1.23.5
zpfrltgup4tujpi1.novalocal        Ready    node                   32s    v1.23.5

进入空间配置好基本环境 新增租户空间等

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME                         STATUS   AGE
default                      Active   13m
elasticsearch                Active   12m
hnc-system                   Active   12m
ingress-nginx                Active   12m
kube-node-lease              Active   13m
kube-public                  Active   13m
kube-system                  Active   13m
kubecube-monitoring          Active   13m
kubecube-project-wordpress   Active   113s
kubecube-system              Active   12m
kubecube-tenant-tenant       Active   2m2s
local-path-storage           Active   12m
wordpresspace                Active   50s

添加一个nginx 测试deployment

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n  wordpresspace
NAME                          READY   STATUS    RESTARTS   AGE
pod/dep-ng-5f6b6f4778-j5vb2   1/1     Running   0          62s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dep-ng   1/1     1            1           62s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/dep-ng-5f6b6f4778   1         1         1       62s

开启全部热插件

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name:         common
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:13Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:01Z
  Resource Version:  6058
  UID:               fc94d067-3712-4907-aca6-48d7f4095314
Spec:
  Component:
    Name:       audit
    Status:     enabled
    Name:       logseer
    Namespace:  logseer
    Pkg Name:   logseer-v1.0.0.tgz
    Status:     enabled
    Env:        clustername: "{{.cluster}}"

    Name:       logagent
    Namespace:  logagent
    Pkg Name:   logagent-v1.0.0.tgz
    Status:     enabled
    Name:       elasticsearch
    Namespace:  elasticsearch
    Pkg Name:   elasticsearch-7.8.1.tgz
    Status:     enabled
    Env:        grafana:
  enabled: false
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://172.31.0.171:31291/api/v1/receive

    Name:       kubecube-monitoring
    Namespace:  kubecube-monitoring
    Pkg Name:   kubecube-monitoring-15.4.12.tgz
    Status:     enabled
    Name:       kubecube-thanos
    Namespace:  kubecube-monitoring
    Pkg Name:   thanos-3.18.0.tgz
    Status:     enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled
Events:       <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name:         pivot-cluster
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:15Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:09Z
  Resource Version:  6066
  UID:               098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
  Component:
    Name:    logseer
    Status:  enabled
    Env:     grafana:
  enabled: true
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://kubecube-thanos-receive:19291/api/v1/receive

    Name:  kubecube-monitoring
    Env:   receive:
  tsdbRetention: 7d
  replicaCount: 1
  replicationFactor: 1

    Name:    kubecube-thanos
    Status:  enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled

配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng

image

点击日志查询 image

查看logseer日志

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer 
NAME                           READY   STATUS    RESTARTS   AGE
pod/logseer-657645d857-w5rv5   1/1     Running   0          6m11s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/logseer   ClusterIP   10.105.118.70   <none>        80/TCP    6m11s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/logseer   1/1     1            1           6m11s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/logseer-657645d857   1         1         1       6m11s

##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpresspace"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "*"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1657675115512,
                "lte" : 1657678715512,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1d",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-07-13 10:22:12.384 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
  at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
  at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
  at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
  at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
  at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
  at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
  at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
  at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
  at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
  at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
  at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
  at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
  at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
  at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
  at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
  at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
  at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

期待回复