search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31k stars 2.95k forks source link

[Bug]: [benchmark][LoadBalance] The Milvus instance idles overnight, and the memory of queryNode rises slowly #23573

Closed wangting0128 closed 1 year ago

wangting0128 commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version:2.2.0-20230419-8d585249
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.7.dev1
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

client argo tasks: fouramf-87ghh，fouramf-concurrent-bzzrx

server:

NAME                                                              READY   STATUS      RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-k9c4t-71-7665-etcd-0                                      1/1     Running     0                 18h     10.104.15.189   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-etcd-1                                      1/1     Running     0                 18h     10.104.20.139   4am-node22   <none>           <none>
fouramf-k9c4t-71-7665-etcd-2                                      1/1     Running     0                 18h     10.104.18.149   4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-milvus-datacoord-54bfb467c7-6ph67           1/1     Running     0                 21h     10.104.15.145   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-milvus-datanode-85db4d867b-gcgsr            1/1     Running     0                 21h     10.104.24.39    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-milvus-indexcoord-786495fb68-cdwrp          1/1     Running     0                 21h     10.104.15.146   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-milvus-indexnode-7c7f98c55-c4b5r            1/1     Running     0                 21h     10.104.21.75    4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-proxy-567f975c76-fhwl5               1/1     Running     0                 21h     10.104.20.107   4am-node22   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querycoord-548dc8895-cwzh5           1/1     Running     0                 21h     10.104.21.74    4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-c2zdn           1/1     Running     0                 21h     10.104.15.147   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-hd4ch           1/1     Running     0                 21h     10.104.4.175    4am-node11   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-p2nlr           1/1     Running     0                 18h     10.104.21.151   4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-pjcjn           1/1     Running     0                 21h     10.104.20.108   4am-node22   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-tf5nt           1/1     Running     0                 18h     10.104.16.144   4am-node21   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-xlk8b           1/1     Running     0                 18h     10.104.18.148   4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-milvus-rootcoord-c896646b7-5vxjm            1/1     Running     0                 21h     10.104.17.81    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-minio-0                                     1/1     Running     0                 21h     10.104.15.151   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-minio-1                                     1/1     Running     0                 21h     10.104.24.42    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-minio-2                                     1/1     Running     0                 21h     10.104.18.77    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-minio-3                                     1/1     Running     0                 21h     10.104.17.83    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-0                             1/1     Running     0                 21h     10.104.23.75    4am-node27   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-1                             1/1     Running     0                 21h     10.104.18.79    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-2                             1/1     Running     0                 21h     10.104.24.44    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-broker-0                             1/1     Running     0                 21h     10.104.17.80    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-proxy-0                              1/1     Running     0                 21h     10.104.18.71    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-recovery-0                           1/1     Running     0                 21h     10.104.18.72    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-0                          1/1     Running     0                 21h     10.104.4.178    4am-node11   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-1                          1/1     Running     0                 21h     10.104.17.85    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-2                          1/1     Running     0                 21h     10.104.24.46    4am-node29   <none>           <none>
截屏2023-04-20 16 28 47 截屏2023-04-20 16 29 39 截屏2023-04-20 16 30 02

the memory of queryNode rises slowly

截屏2023-04-20 16 30 18 截屏2023-04-20 16 30 30

client:

截屏2023-04-20 16 27 09 截屏2023-04-20 16 28 20

Expected Behavior

No response

Steps To Reproduce

1、deploy Milvus Cluster
2、client：
   1. create collection
   2. build index of IVF_SQ8
   3. insert 20m vectors
   4. flush
   5. rebuild IVF_SQ8 index
   6. load
   7. locust search <- concurrent
3、concurrent insert to the same collection
4、scale out queryNode from 3 to 4
5、concurrent search to the same collection and scale out queryNode from 4 to 6
6、concurrent 5 clients and the same as step 2
7、The Milvus instance idles overnight 《- the memory of queryNode rises slowly

Milvus Log

No response

Anything else?

No response

yanliang567 commented 1 year ago

/unassign

wangting0128 commented 1 year ago

same phenomenon, different scene

client argo tasks: fouramf-stnjq, fouramf-concurrent-q2phm

Steps To Reproduce

1、deploy Milvus Cluster
2、client：
   1. create collection
   2. build index of IVF_SQ8
   3. insert 20m vectors
   4. flush
   5. rebuild IVF_SQ8 index
   6. load
   7. locust search <- concurrent
3、concurrent insert to the same collection
4、scale out queryNode from 4 to 5
5、concurrent search to the same collection and scale out queryNode from 5 to 6
6、concurrent 2 clients and the same as step 2, one collection set replica=2, dataset size=20m; another one set replica=3, dataset size=10m
7、The Milvus instance idles overnight 《- the memory of queryNode rises slowly

server:

NAME                                                              READY   STATUS      RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-l2mgq-61-3057-etcd-0                                      1/1     Running     0                 18h     10.104.23.124   4am-node27   <none>           <none>
fouramf-l2mgq-61-3057-etcd-1                                      1/1     Running     0                 18h     10.104.15.190   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-etcd-2                                      1/1     Running     0                 18h     10.104.16.143   4am-node21   <none>           <none>
fouramf-l2mgq-61-3057-milvus-datacoord-756fd4fb99-2v5d8           1/1     Running     0                 21h     10.104.21.77    4am-node24   <none>           <none>
fouramf-l2mgq-61-3057-milvus-datanode-7c5fd9f69d-rjsdq            1/1     Running     1 (21h ago)       21h     10.104.21.78    4am-node24   <none>           <none>
fouramf-l2mgq-61-3057-milvus-indexcoord-7b96f6d694-8xlzf          1/1     Running     1 (21h ago)       21h     10.104.15.153   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-milvus-indexnode-54589b88f-twbvv            1/1     Running     0                 21h     10.104.23.76    4am-node27   <none>           <none>
fouramf-l2mgq-61-3057-milvus-proxy-7579976fd6-926pp               1/1     Running     1 (21h ago)       21h     10.104.21.76    4am-node24   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querycoord-7f6dd89cc7-p4bnb          1/1     Running     1 (21h ago)       21h     10.104.15.152   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-29v2j           1/1     Running     0                 21h     10.104.9.123    4am-node14   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-c5kbq           1/1     Running     0                 21h     10.104.17.89    4am-node23   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-dstqm           1/1     Running     0                 21h     10.104.20.112   4am-node22   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-qpwsq           1/1     Running     0                 18h     10.104.16.145   4am-node21   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-t88tb           1/1     Running     0                 18h     10.104.15.188   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-milvus-querynode-5586c588dc-x8k4g           1/1     Running     0                 21h     10.104.18.80    4am-node25   <none>           <none>
fouramf-l2mgq-61-3057-milvus-rootcoord-b58b4fc45-cvbgm            1/1     Running     0                 21h     10.104.17.86    4am-node23   <none>           <none>
fouramf-l2mgq-61-3057-minio-0                                     1/1     Running     0                 21h     10.104.24.50    4am-node29   <none>           <none>
fouramf-l2mgq-61-3057-minio-1                                     1/1     Running     0                 21h     10.104.4.180    4am-node11   <none>           <none>
fouramf-l2mgq-61-3057-minio-2                                     1/1     Running     0                 21h     10.104.16.79    4am-node21   <none>           <none>
fouramf-l2mgq-61-3057-minio-3                                     1/1     Running     0                 21h     10.104.15.157   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-bookie-0                             1/1     Running     0                 21h     10.104.24.51    4am-node29   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-bookie-1                             1/1     Running     0                 21h     10.104.4.183    4am-node11   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-bookie-2                             1/1     Running     0                 21h     10.104.1.41     4am-node10   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-broker-0                             1/1     Running     0                 21h     10.104.5.55     4am-node12   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-proxy-0                              1/1     Running     0                 21h     10.104.5.54     4am-node12   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-recovery-0                           1/1     Running     0                 21h     10.104.23.77    4am-node27   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-zookeeper-0                          1/1     Running     0                 21h     10.104.16.82    4am-node21   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-zookeeper-1                          1/1     Running     0                 21h     10.104.15.159   4am-node20   <none>           <none>
fouramf-l2mgq-61-3057-pulsar-zookeeper-2                          1/1     Running     0                 21h     10.104.19.109   4am-node28   <none>           <none>
截屏2023-04-20 16 49 58 截屏2023-04-20 16 55 11 截屏2023-04-20 16 55 29 截屏2023-04-20 16 55 57 截屏2023-04-20 16 56 06

client:

截屏2023-04-20 16 53 28

image

wangting0128 commented 1 year ago

server:

NAME                                                              READY   STATUS      RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-k9c4t-71-7665-etcd-0                                      1/1     Running     0                 37h     10.104.15.189   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-etcd-1                                      1/1     Running     0                 37h     10.104.20.139   4am-node22   <none>           <none>
fouramf-k9c4t-71-7665-etcd-2                                      1/1     Running     0                 37h     10.104.18.149   4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-milvus-datacoord-54bfb467c7-6ph67           1/1     Running     0                 40h     10.104.15.145   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-milvus-datanode-85db4d867b-gcgsr            1/1     Running     0                 40h     10.104.24.39    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-milvus-indexcoord-786495fb68-cdwrp          1/1     Running     0                 40h     10.104.15.146   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-milvus-indexnode-7c7f98c55-c4b5r            1/1     Running     0                 40h     10.104.21.75    4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-proxy-567f975c76-fhwl5               1/1     Running     0                 40h     10.104.20.107   4am-node22   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querycoord-548dc8895-cwzh5           1/1     Running     0                 40h     10.104.21.74    4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-hd4ch           1/1     Running     6 (13h ago)       40h     10.104.4.175    4am-node11   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-p2nlr           1/1     Running     0                 37h     10.104.21.151   4am-node24   <none>           <none>
fouramf-k9c4t-71-7665-milvus-querynode-775467c8fc-tf5nt           1/1     Running     0                 37h     10.104.16.144   4am-node21   <none>           <none>
fouramf-k9c4t-71-7665-milvus-rootcoord-c896646b7-5vxjm            1/1     Running     0                 40h     10.104.17.81    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-minio-0                                     1/1     Running     0                 40h     10.104.15.151   4am-node20   <none>           <none>
fouramf-k9c4t-71-7665-minio-1                                     1/1     Running     0                 40h     10.104.24.42    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-minio-2                                     1/1     Running     0                 40h     10.104.18.77    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-minio-3                                     1/1     Running     0                 40h     10.104.17.83    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-0                             1/1     Running     0                 40h     10.104.23.75    4am-node27   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-1                             1/1     Running     0                 40h     10.104.18.79    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-bookie-2                             1/1     Running     0                 40h     10.104.24.44    4am-node29   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-broker-0                             1/1     Running     0                 40h     10.104.17.80    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-proxy-0                              1/1     Running     0                 40h     10.104.18.71    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-recovery-0                           1/1     Running     0                 40h     10.104.18.72    4am-node25   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-0                          1/1     Running     0                 40h     10.104.4.178    4am-node11   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-1                          1/1     Running     0                 40h     10.104.17.85    4am-node23   <none>           <none>
fouramf-k9c4t-71-7665-pulsar-zookeeper-2                          1/1     Running     0                 40h     10.104.24.46    4am-node29   <none>           <none>

Milvus idles during: 2022-04-20 22:22:00 ~ 2022-04-21 11:00:00

memory rises ~5G

截屏2023-04-21 11 02 18 截屏2023-04-21 11 03 48 截屏2023-04-21 11 04 13 截屏2023-04-21 11 04 34 截屏2023-04-21 11 04 45 截屏2023-04-21 11 05 18 截屏2023-04-21 11 05 28
aoiasd commented 1 year ago

/assign

elstic commented 1 year ago

case : test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_replica2_cluster After running the test and waiting a few hours without doing anything, querynode memory rises.

server : 

fouramf-stbsk-99-2262-etcd-0                                      1/1     Running            0                3m54s   10.104.16.14    4am-node21   <none>           <none>
fouramf-stbsk-99-2262-etcd-1                                      1/1     Running            0                3m54s   10.104.24.70    4am-node29   <none>           <none>
fouramf-stbsk-99-2262-etcd-2                                      1/1     Running            0                3m54s   10.104.21.94    4am-node24   <none>           <none>
fouramf-stbsk-99-2262-milvus-datacoord-6c9dd7549d-2w2sz           1/1     Running            0                3m54s   10.104.22.195   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-milvus-datanode-76d869bff6-brgw6            1/1     Running            0                3m54s   10.104.22.193   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-milvus-indexcoord-8696cddfbf-s7zhr          1/1     Running            0                3m54s   10.104.22.189   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-milvus-indexnode-5d7b5fffbb-mhmfr           1/1     Running            0                3m54s   10.104.5.11     4am-node12   <none>           <none>
fouramf-stbsk-99-2262-milvus-proxy-54bf8c597c-dxb8z               1/1     Running            0                3m54s   10.104.24.65    4am-node29   <none>           <none>
fouramf-stbsk-99-2262-milvus-querycoord-658d8f4886-ckpkb          1/1     Running            0                3m54s   10.104.22.190   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-milvus-querynode-68fcc5985d-9k8hw           1/1     Running            0                3m54s   10.104.6.198    4am-node13   <none>           <none>
fouramf-stbsk-99-2262-milvus-querynode-68fcc5985d-ftb4p           1/1     Running            0                3m54s   10.104.22.191   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-milvus-rootcoord-8dc7bd4fb-4sxnj            1/1     Running            0                3m54s   10.104.22.188   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-minio-0                                     1/1     Running            0                3m54s   10.104.21.91    4am-node24   <none>           <none>
fouramf-stbsk-99-2262-minio-1                                     1/1     Running            0                3m54s   10.104.24.67    4am-node29   <none>           <none>
fouramf-stbsk-99-2262-minio-2                                     1/1     Running            0                3m54s   10.104.4.137    4am-node11   <none>           <none>
fouramf-stbsk-99-2262-minio-3                                     1/1     Running            0                3m54s   10.104.23.169   4am-node27   <none>           <none>
fouramf-stbsk-99-2262-pulsar-bookie-0                             1/1     Running            0                3m54s   10.104.16.17    4am-node21   <none>           <none>
fouramf-stbsk-99-2262-pulsar-bookie-1                             1/1     Running            0                3m54s   10.104.24.72    4am-node29   <none>           <none>
fouramf-stbsk-99-2262-pulsar-bookie-2                             1/1     Running            0                3m53s   10.104.4.140    4am-node11   <none>           <none>
fouramf-stbsk-99-2262-pulsar-bookie-init-f48rh                    0/1     Completed          0                3m54s   10.104.21.88    4am-node24   <none>           <none>
fouramf-stbsk-99-2262-pulsar-broker-0                             1/1     Running            0                3m54s   10.104.4.135    4am-node11   <none>           <none>
fouramf-stbsk-99-2262-pulsar-proxy-0                              1/1     Running            0                3m54s   10.104.24.64    4am-node29   <none>           <none>
fouramf-stbsk-99-2262-pulsar-pulsar-init-98z8n                    0/1     Completed          0                3m54s   10.104.22.187   4am-node26   <none>           <none>
fouramf-stbsk-99-2262-pulsar-recovery-0                           1/1     Running            0                3m54s   10.104.23.165   4am-node27   <none>           <none>
fouramf-stbsk-99-2262-pulsar-zookeeper-0                          1/1     Running            0                3m54s   10.104.21.92    4am-node24   <none>           <none>
fouramf-stbsk-99-2262-pulsar-zookeeper-1                          1/1     Running            0                3m15s   10.104.23.171   4am-node27   <none>           <none>
fouramf-stbsk-99-2262-pulsar-zookeeper-2                          1/1     Running            0                2m40s   10.104.16.21    4am-node21   <none>           <none>

image

image

xiaofan-luan commented 1 year ago

should a analysis on the weird increase

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

wangting0128 commented 1 year ago

keep an eye on it

elstic commented 1 year ago

case: test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_kafka_cluster image: master-20230905-fb0705df

querynode memory up at 28h

image

querynode memory went from 1.19g to 5.54g

server:

fouramf-9jq2k-13-8541-etcd-0                                      1/1     Running     0               2d3h    10.104.17.83    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-etcd-1                                      1/1     Running     0               2d3h    10.104.24.33    4am-node29   <none>           <none>
fouramf-9jq2k-13-8541-etcd-2                                      1/1     Running     1               2d3h    10.104.12.107   4am-node17   <none>           <none>
fouramf-9jq2k-13-8541-kafka-0                                     1/1     Running     2 (2d3h ago)    2d3h    10.104.12.108   4am-node17   <none>           <none>
fouramf-9jq2k-13-8541-kafka-1                                     1/1     Running     1 (2d3h ago)    2d3h    10.104.24.37    4am-node29   <none>           <none>
fouramf-9jq2k-13-8541-kafka-2                                     1/1     Running     2 (2d3h ago)    2d3h    10.104.17.87    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-milvus-datacoord-fcddf56f6-qmhxj            1/1     Running     0               2d3h    10.104.17.80    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-milvus-datanode-76bb646c68-tb54f            1/1     Running     0               2d3h    10.104.4.102    4am-node11   <none>           <none>
fouramf-9jq2k-13-8541-milvus-indexcoord-7f6d75795c-sknc7          1/1     Running     0               2d3h    10.104.17.78    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-milvus-indexnode-5457c4f4b8-zlgtr           1/1     Running     0               2d3h    10.104.14.88    4am-node18   <none>           <none>
fouramf-9jq2k-13-8541-milvus-proxy-8847b7bc5-95jzf                1/1     Running     0               2d3h    10.104.14.85    4am-node18   <none>           <none>
fouramf-9jq2k-13-8541-milvus-querycoord-df8bf659d-ftr4h           1/1     Running     0               2d3h    10.104.4.100    4am-node11   <none>           <none>
fouramf-9jq2k-13-8541-milvus-querynode-57d4f579bd-vvdmz           1/1     Running     0               2d3h    10.104.12.101   4am-node17   <none>           <none>
fouramf-9jq2k-13-8541-milvus-rootcoord-858f9dc8c8-qs99s           1/1     Running     0               2d3h    10.104.4.101    4am-node11   <none>           <none>
fouramf-9jq2k-13-8541-minio-0                                     1/1     Running     0               2d3h    10.104.24.31    4am-node29   <none>           <none>
fouramf-9jq2k-13-8541-minio-1                                     1/1     Running     0               2d3h    10.104.17.93    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-minio-2                                     1/1     Running     0               2d3h    10.104.9.76     4am-node14   <none>           <none>
fouramf-9jq2k-13-8541-minio-3                                     1/1     Running     0               2d3h    10.104.1.173    4am-node10   <none>           <none>
fouramf-9jq2k-13-8541-zookeeper-0                                 1/1     Running     0               2d3h    10.104.12.106   4am-node17   <none>           <none>
fouramf-9jq2k-13-8541-zookeeper-1                                 1/1     Running     0               2d3h    10.104.17.86    4am-node23   <none>           <none>
fouramf-9jq2k-13-8541-zookeeper-2                                 1/1     Running     0               2d3h    10.104.24.36    4am-node29   <none>           <none>
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.