search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.96k stars 2.87k forks source link

[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

Open wangting0128 opened 3 days ago

wangting0128 commented 3 days ago

Is there an existing issue for this?

Environment

- Milvus version:master-20241013-d230b91b-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-concurrent-kd529 test case name: test_bitmap_locust_dql_dml_standalone

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-54-5077-etcd-0                                             1/1     Running            0               41m     10.104.17.251   4am-node23   <none>           <none>
fouram-54-5077-milvus-standalone-7d64cf9cf6-ftfbs                 0/1     CrashLoopBackOff   7 (68s ago)     41m     10.104.25.169   4am-node30   <none>           <none>
fouram-54-5077-minio-6888fdd5b4-4dhpx                             1/1     Running            0               41m     10.104.17.250   4am-node23   <none>           <none>
截屏2024-10-14 11 52 44

fouram-54-5077-milvus-standalone-7d64cf9cf6-ftfbs-panic.log

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64 autoID`
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. verify DQL & DML requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector'
                BITMAP: all scalar fields
            3. insert 2 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - search
                - query
                - hybrid_search
                - load
                - insert
                - delete: delete all inserted data
                - flush: ignore RateLimiter

Milvus Log

No response

Anything else?

server config:

{
     "standalone": {
          "resources": {
               "limits": {
                    "cpu": "16.0",
                    "memory": "64Gi"
               },
               "requests": {
                    "cpu": "9.0",
                    "memory": "33Gi"
               }
          }
     },
     "cluster": {
          "enabled": false
     },
     "etcd": {
          "replicaCount": 1,
          "metrics": {
               "enabled": true,
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "minio": {
          "mode": "standalone",
          "metrics": {
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "pulsar": {
          "enabled": false
     },
     "metrics": {
          "serviceMonitor": {
               "enabled": true
          }
     },
     "log": {
          "level": "debug"
     },
     "extraConfigFiles": {
          "user.yaml": "queryNode:\n  segcore:\n    multipleChunkedEnable: true\n"
     },
     "image": {
          "all": {
               "repository": "harbor.milvus.io/milvus/milvus",
               "tag": "master-20241013-d230b91b-amd64"
          }
     }
}

client config:

{
     "dataset_params": {
          "metric_type": "L2",
          "dim": 128,
          "max_length": 100,
          "scalars_index": {
               "int8_1": {
                    "index_type": "BITMAP"
               },
               "int16_1": {
                    "index_type": "BITMAP"
               },
               "int32_1": {
                    "index_type": "BITMAP"
               },
               "int64_1": {
                    "index_type": "BITMAP"
               },
               "varchar_1": {
                    "index_type": "BITMAP"
               },
               "bool_1": {
                    "index_type": "BITMAP"
               },
               "array_int8_1": {
                    "index_type": "BITMAP"
               },
               "array_int16_1": {
                    "index_type": "BITMAP"
               },
               "array_int32_1": {
                    "index_type": "BITMAP"
               },
               "array_int64_1": {
                    "index_type": "BITMAP"
               },
               "array_varchar_1": {
                    "index_type": "BITMAP"
               },
               "array_bool_1": {
                    "index_type": "BITMAP"
               }
          },
          "vectors_index": {
               "sparse_float_vector": {
                    "index_type": "SPARSE_INVERTED_INDEX",
                    "index_param": {
                         "drop_ratio_build": 0.2
                    },
                    "metric_type": "IP"
               }
          },
          "scalars_params": {
               "array_int8_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -128,
                                   128
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int16_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -200,
                                   200
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int32_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope",
                              "specify_range": [
                                   -300,
                                   300
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int64_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "fixed_value_range",
                              "specify_range": [
                                   -400,
                                   432
                              ],
                              "batch": 50,
                              "max_capacity": 13
                         }
                    }
               },
               "array_varchar_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -1500,
                                   1500
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_bool_1": {
                    "params": {
                         "max_capacity": 13
                    }
               },
               "int8_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -128,
                                   128
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int16_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -200,
                                   200
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int32_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope",
                              "specify_range": [
                                   -300,
                                   300
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int64_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "fixed_value_range",
                              "specify_range": [
                                   -400,
                                   432
                              ],
                              "batch": 50,
                              "max_capacity": 13
                         }
                    }
               },
               "varchar_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -1500,
                                   1500
                              ],
                              "max_capacity": 13
                         }
                    }
               }
          },
          "dataset_name": "sift",
          "dataset_size": 2000000,
          "ni_per": 5000
     },
     "collection_params": {
          "other_fields": [
               "sparse_float_vector",
               "int8_1",
               "int16_1",
               "int32_1",
               "int64_1",
               "varchar_1",
               "bool_1",
               "array_int8_1",
               "array_int16_1",
               "array_int32_1",
               "array_int64_1",
               "array_varchar_1",
               "array_bool_1"
          ],
          "shards_num": 1,
          "auto_id": true
     },
     "resource_groups_params": {
          "reset": false
     },
     "database_user_params": {
          "reset_rbac": false,
          "reset_db": false
     },
     "index_params": {
          "index_type": "IVF_SQ8",
          "index_param": {
               "nlist": 1024
          }
     },
     "concurrent_params": {
          "concurrent_number": [
               20
          ],
          "during_time": "3h",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 10,
                    "search_param": {
                         "nprobe": 16
                    },
                    "expr": "int8_1 == 100",
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "output_fields": [
                         "id",
                         "float_vector",
                         "int64_1"
                    ],
                    "ignore_growing": false,
                    "group_by_field": null,
                    "timeout": 60,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "nq": 10
                    }
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "ids": null,
                    "expr": "int64_1 > -1",
                    "output_fields": [
                         "*"
                    ],
                    "offset": null,
                    "limit": 10,
                    "ignore_growing": false,
                    "partition_names": null,
                    "timeout": 60,
                    "consistency_level": null,
                    "random_data": false,
                    "random_count": 0,
                    "random_range": [
                         0,
                         1
                    ],
                    "field_name": "id",
                    "field_type": "int64",
                    "check_task": "check_query_output",
                    "check_items": {
                         "expect_length": 10
                    }
               }
          },
          {
               "type": "hybrid_search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 1,
                    "reqs": [
                         {
                              "search_param": {
                                   "nprobe": 128
                              },
                              "anns_field": "float_vector",
                              "expr": "(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, 1)) || ((varchar_1 like \"1%\") and (bool_1 == True))",
                              "top_k": 100
                         },
                         {
                              "search_param": {
                                   "drop_ratio_search": 0.1
                              },
                              "anns_field": "sparse_float_vector",
                              "expr": "not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, 1])"
                         }
                    ],
                    "rerank": {
                         "RRFRanker": []
                    },
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "timeout": 120,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "output_fields": [
                              "sparse_float_vector",
                              "int8_1",
                              "int16_1",
                              "int32_1",
                              "int64_1",
                              "varchar_1",
                              "bool_1",
                              "array_int8_1",
                              "array_int16_1",
                              "array_int32_1",
                              "array_int64_1",
                              "array_varchar_1",
                              "array_bool_1",
                              "id",
                              "float_vector"
                         ],
                         "nq": 10
                    }
               }
          },
          {
               "type": "load",
               "weight": 1,
               "params": {
                    "replica_number": 1,
                    "timeout": 180,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "insert",
               "weight": 1,
               "params": {
                    "nb": 10,
                    "timeout": 30,
                    "random_id": true,
                    "random_vector": true,
                    "varchar_filled": false,
                    "start_id": 2000000,
                    "shuffle_id": false,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "delete",
               "weight": 1,
               "params": {
                    "expr": "",
                    "delete_length": 10,
                    "timeout": 30,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "flush",
               "weight": 1,
               "params": {
                    "timeout": 600,
                    "check_task": "check_ignore_expected_errors",
                    "check_items": [
                         {
                              "message": "request is rejected by grpc RateLimiter middleware, please retry later"
                         },
                         {
                              "message": "wait for flush timeout"
                         }
                    ]
               }
          }
     ]
}
wangting0128 commented 3 days ago

Running e2e test cases L0,L1,L2, has the same panic

argo task: fouramf-e2e-multi-chunked-enable

server:

NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-multi-chunked-enable-etcd-0                           1/1     Running            0                43m     10.104.30.212   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-etcd-1                           1/1     Running            0                43m     10.104.18.115   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-etcd-2                           1/1     Running            0                43m     10.104.23.37    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-datanode-995b69b6bjkmxc   1/1     Running            2 (43m ago)      43m     10.104.34.7     4am-node37   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-indexnode-594b66d7vn2ml   1/1     Running            2 (43m ago)      43m     10.104.16.203   4am-node21   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-mixcoord-f4844558-5jkzn   1/1     Running            2 (43m ago)      43m     10.104.16.202   4am-node21   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-proxy-7f55b5ddcb-r5749    1/1     Running            2 (43m ago)      43m     10.104.32.153   4am-node39   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-querynode-6bc48bdbgdrw9   0/1     Running            11 (5m12s ago)   43m     10.104.32.152   4am-node39   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-0                          1/1     Running            0                43m     10.104.23.34    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-1                          1/1     Running            0                43m     10.104.30.211   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-2                          1/1     Running            0                43m     10.104.20.189   4am-node22   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-3                          1/1     Running            0                43m     10.104.18.119   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-0                  1/1     Running            0                43m     10.104.23.35    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-1                  1/1     Running            0                43m     10.104.30.213   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-2                  1/1     Running            0                43m     10.104.20.193   4am-node22   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-init-2k8gh         0/1     Completed          0                43m     10.104.23.27    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-broker-0                  1/1     Running            0                43m     10.104.6.198    4am-node13   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-proxy-0                   1/1     Running            0                43m     10.104.5.21     4am-node12   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-pulsar-init-zgkk5         0/1     Completed          0                43m     10.104.30.204   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-recovery-0                1/1     Running            0                43m     10.104.6.197    4am-node13   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-0               1/1     Running            0                43m     10.104.18.113   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-1               1/1     Running            0                43m     10.104.23.39    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-2               1/1     Running            0                42m     10.104.17.254   4am-node23   <none>           <none>
截屏2024-10-14 11 57 16
wangting0128 commented 2 days ago

different case, same panic

argo task: fouramf-concurrent-kd529 test case name: test_inverted_locust_hnsw_diskann_dml_dql_cluster image: master-20241013-d230b91b-amd64

server:

NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-10-2077-etcd-0                                             1/1     Running            0                3h12m   10.104.23.12    4am-node27   <none>           <none>
fouram-10-2077-etcd-1                                             1/1     Running            0                3h12m   10.104.18.99    4am-node25   <none>           <none>
fouram-10-2077-etcd-2                                             1/1     Running            0                3h12m   10.104.30.203   4am-node38   <none>           <none>
fouram-10-2077-milvus-datanode-74b5f89c87-jx7sn                   1/1     Running            2 (3h12m ago)    3h12m   10.104.14.33    4am-node18   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-64j52                   1/1     Running            2 (3h12m ago)    3h12m   10.104.19.163   4am-node28   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-k9d4t                   1/1     Running            2 (3h12m ago)    3h12m   10.104.1.125    4am-node10   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-k9rwr                   1/1     Running            2 (3h12m ago)    3h12m   10.104.17.252   4am-node23   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-n48bp                   1/1     Running            2 (3h12m ago)    3h12m   10.104.33.34    4am-node36   <none>           <none>
fouram-10-2077-milvus-mixcoord-5c988f665f-2s9n4                   1/1     Running            2 (3h12m ago)    3h12m   10.104.14.32    4am-node18   <none>           <none>
fouram-10-2077-milvus-proxy-74d98c7bff-wngqg                      1/1     Running            2 (3h12m ago)    3h12m   10.104.14.31    4am-node18   <none>           <none>
fouram-10-2077-milvus-querynode-6ccb56788d-sxb7d                  0/1     CrashLoopBackOff   28 (4m37s ago)   3h12m   10.104.24.234   4am-node29   <none>           <none>
fouram-10-2077-milvus-querynode-6ccb56788d-xdhsz                  0/1     CrashLoopBackOff   30 (3m1s ago)    3h12m   10.104.9.115    4am-node14   <none>           <none>
fouram-10-2077-minio-0                                            1/1     Running            0                3h12m   10.104.18.101   4am-node25   <none>           <none>
fouram-10-2077-minio-1                                            1/1     Running            0                3h12m   10.104.30.201   4am-node38   <none>           <none>
fouram-10-2077-minio-2                                            1/1     Running            0                3h12m   10.104.23.13    4am-node27   <none>           <none>
fouram-10-2077-minio-3                                            1/1     Running            0                3h12m   10.104.20.183   4am-node22   <none>           <none>
fouram-10-2077-pulsar-bookie-0                                    1/1     Running            0                3h12m   10.104.30.202   4am-node38   <none>           <none>
fouram-10-2077-pulsar-bookie-1                                    1/1     Running            0                3h12m   10.104.18.104   4am-node25   <none>           <none>
fouram-10-2077-pulsar-bookie-2                                    1/1     Running            0                3h12m   10.104.20.185   4am-node22   <none>           <none>
fouram-10-2077-pulsar-bookie-init-z659n                           0/1     Completed          0                3h12m   10.104.6.190    4am-node13   <none>           <none>
fouram-10-2077-pulsar-broker-0                                    1/1     Running            0                3h12m   10.104.4.220    4am-node11   <none>           <none>
fouram-10-2077-pulsar-proxy-0                                     1/1     Running            0                3h12m   10.104.4.219    4am-node11   <none>           <none>
fouram-10-2077-pulsar-pulsar-init-rq5tr                           0/1     Completed          0                3h12m   10.104.6.191    4am-node13   <none>           <none>
fouram-10-2077-pulsar-recovery-0                                  1/1     Running            0                3h12m   10.104.6.192    4am-node13   <none>           <none>
fouram-10-2077-pulsar-zookeeper-0                                 1/1     Running            0                3h12m   10.104.30.200   4am-node38   <none>           <none>
fouram-10-2077-pulsar-zookeeper-1                                 1/1     Running            0                3h11m   10.104.23.17    4am-node27   <none>           <none>
fouram-10-2077-pulsar-zookeeper-2                                 1/1     Running            0                3h11m   10.104.18.109   4am-node25   <none>           <none>
截屏2024-10-14 14 24 54
xiaofan-luan commented 2 days ago

/assign @sunby

let's enable this feature by default so we can find more bug

sunby commented 2 days ago

/assign @sunby

let's enable this feature by default so we can find more bug

yes, we plan to enable this after some stability tests

sunby commented 2 days ago

I will build a debug image and try to reproduce it.

wangting0128 commented 1 day ago

e2e test verification failed

argo task：fouramf-e2e-sby-fix image: master-20241016-b2037c95-amd64

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-sby-fix-etcd-0                                        1/1     Running            0               23m     10.104.19.120   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-etcd-1                                        1/1     Running            0               23m     10.104.32.178   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-etcd-2                                        1/1     Running            0               23m     10.104.34.119   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-milvus-datanode-66fcb7bd44-27csb              1/1     Running            3 (22m ago)     23m     10.104.32.169   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-milvus-indexnode-f79868cf7-gs8cx              1/1     Running            3 (22m ago)     23m     10.104.19.98    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-mixcoord-55df88fc46-w2vsf              1/1     Running            4 (18m ago)     23m     10.104.19.96    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-proxy-6894d45f48-778l6                 1/1     Running            4 (18m ago)     23m     10.104.19.100   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-querynode-76bc48c667-vdffr             0/1     CrashLoopBackOff   9 (3m16s ago)   23m     10.104.32.168   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-0                                       1/1     Running            0               23m     10.104.19.122   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-minio-1                                       1/1     Running            0               23m     10.104.32.180   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-2                                       1/1     Running            0               23m     10.104.34.120   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-minio-3                                       1/1     Running            0               23m     10.104.20.105   4am-node22   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-0                               1/1     Running            0               23m     10.104.19.121   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-1                               1/1     Running            0               23m     10.104.32.179   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-2                               1/1     Running            0               23m     10.104.26.217   4am-node32   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-init-4f5hb                      0/1     Completed          0               23m     10.104.19.102   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-broker-0                               1/1     Running            0               23m     10.104.32.171   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-proxy-0                                1/1     Running            0               23m     10.104.19.97    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-pulsar-init-k4bxx                      0/1     Completed          0               23m     10.104.19.99    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-recovery-0                             1/1     Running            0               23m     10.104.19.101   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-0                            1/1     Running            0               23m     10.104.19.119   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-1                            1/1     Running            0               22m     10.104.32.182   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-2                            1/1     Running            0               21m     10.104.34.125   4am-node37   <none>           <none>
截屏2024-10-16 12 12 50