[Bug]: [benchmark][standalone]With the same machine configuration, milvus image and pymilvus version, 50% difference in latency both in search and query

[jingkl]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.1.0-20220711-1c74adcf
- Deployment mode(standalone or cluster):standalone
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus2.1.0dev98
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1.The qa-node017 machine has the same configuration as the 4am-node12 machine

2. Milvus image and pymilvus version are the same

server-instance benchmark-backup-nhc5d-1 server-configmap server-single-2c8m-n17 client-configmap client-random-locust-search-1m server ：

benchmark-backup-pwf8s-1-etcd-0                               1/1     Running       0          2m3s    10.97.17.199   qa-node014.zilliz.local   <none>           <none>
benchmark-backup-pwf8s-1-milvus-standalone-549f978b76-wwn9g   1/1     Running       0          2m3s    10.97.18.206   qa-node017.zilliz.local   <none>           <none>
benchmark-backup-pwf8s-1-minio-9cdc7d978-bnth4                1/1     Running       0          2m3s    10.97.20.155   qa-node018.zilliz.local   <none>           <none>

Search latency：

query :

server-instance fouram-w96gh-1 server-configmap server-single-2c8m client-configmap client-random-locust-search-1m

server ：

fouram-w96gh-1-etcd-0                               1/1     Running   0             109s   10.104.9.34    4am-node14   <none>           <none>
fouram-w96gh-1-milvus-standalone-75d4cc464d-lwk86   1/1     Running   0             109s   10.104.5.25    4am-node12   <none>           <none>
fouram-w96gh-1-minio-6dbd6596fb-vvt9c               1/1     Running   0             109s   10.104.4.102   4am-node11   <none>           <none>

Search latency：

query:

Expected Behavior

No response

Steps To Reproduce

1、create collection
2、create index of ivf_sq8
3、insert 1m million vectors
4、flush collection
5、build index with the same params
6、load collection
7、locust concurrent: query<-search, load, get<-query

Milvus Log

No response

Anything else?

client-random-locust-search-1m：

config.yaml: |
    locust_random_performance:
      collections:
        -
          milvus:
            cache_config.insert_buffer_size: 2GB
            engine_config.use_blas_threshold: 1100
            wal_enable: true
          collection_name: sift_1m_128_l2
          ni_per: 50000
          build_index: true
          index_type: ivf_sq8
          index_param:
            nlist: 1024
          task:
            types:
              -
                type: query
                weight: 20
                params:
                  top_k: 10
                  nq: 10
                  search_param:
                    nprobe: 16
              -
                type: load
                weight: 1
              -
                type: get
                weight: 2
                params:
                  ids_length: 10
            connection_num: 1
            clients_num: 20
            spawn_rate: 2
            during_time: 2h

[yanliang567]

/assign @czs007 /unassign

[xiaofan-luan]

search should definitely slower than query, so we are saying query is slower than search?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[jingkl]

/reopen

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.