alwayslove2013
commented
7 months ago @gaowayne maybe you can try it first
pip install --ignore-installed setuptools Closed gaowayne closed 6 months ago
alwayslove2013
commented
7 months ago @gaowayne maybe you can try it first
pip install --ignore-installed setuptools 
gaowayne
commented
7 months ago @gaowayne maybe you can try it first
pip install --ignore-installed setuptools
man how can I make it point to my nvme device during run vectordb, could you please suggest which test can stress NVMe write and read most? :)
gaowayne
commented
7 months ago I got GUI start, but when I run Milvus, it shows below exception.
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>
2024-02-29 21:03:57,013 | INFO: [3/3] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus'}, drop_old=True (interface.py:167) (939711)
2024-02-29 21:04:27,016 | WARNING: pre run case error: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)> (task_runner.py:92) (939711)
2024-02-29 21:04:27,016 | WARNING: [3/3] case {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus'} failed to run, reason=<MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)> (interface.py:187) (939711)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/interface.py", line 168, in _async_task_v2
case_res.metrics = runner.run(drop_old)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 96, in run
self._pre_run(drop_old)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 93, in _pre_run
raise e from None
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 86, in _pre_run
self.init_db(drop_old)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 77, in init_db
self.db = db_cls(
^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/clients/milvus/milvus.py", line 43, in __init__
connections.connect(**self.db_config, timeout=30)
File "/usr/local/lib/python3.11/site-packages/pymilvus/orm/connections.py", line 414, in connect
connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
File "/usr/local/lib/python3.11/site-packages/pymilvus/orm/connections.py", line 365, in connect_milvus
gh._wait_for_channel_ready(timeout=timeout)
File "/usr/local/lib/python3.11/site-packages/pymilvus/client/grpc_handler.py", line 146, in _wait_for_channel_ready
raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>
2024-02-29 21:04:27,019 | INFO |Task summary: run_id=77945, task_label=2024022921 (models.py:285)
2024-02-29 21:04:27,019 | INFO |DB | db_label case label | load_dur qps latency(p99) recall max_load_count | label (models.py:285)
2024-02-29 21:04:27,019 | INFO |------ | -------- ------------------- ---------- | ----------- ---------- --------------- ------------- -------------- | ----- (models.py:285)
2024-02-29 21:04:27,019 | INFO |Milvus | Performance768D100M 2024022921 | 0.0 0.0 0.0 0.0 0 | x (models.py:285)
2024-02-29 21:04:27,019 | INFO |Milvus | Performance1536D5M 2024022921 | 0.0 0.0 0.0 0.0 0 | x (models.py:285)
2024-02-29 21:04:27,019 | INFO |Milvus | CapacityDim960 2024022921 | 0.0 0.0 0.0 0.0 0 | x (models.py:285)
2024-02-29 21:04:27,020 | INFO: write results to disk /usr/local/lib/python3.11/site-packages/vectordb_bench/results/Milvus/result_20240229_2024022921_milvus.json (models.py:143) (939711)
2024-02-29 21:04:27,022 | INFO: Succes to finish task: label=2024022921, run_id=77945d9529244d75b77c784a3e6a46f0 (interface.py:207) (939711)
2024-02-29 21:06:38,230 | INFO: generated uuid for the tasks: fe4bb641681a4899a2598c48fdee9497 (interface.py:69) (939332)
2024-02-29 21:06:38,234 | INFO | DB | CaseType Dataset Filter | task_label (task_runner.py:288)
2024-02-29 21:06:38,234 | INFO | ----------- | ------------ -------------------- ------- | ------- (task_runner.py:288)
2024-02-29 21:06:38,234 | INFO | Milvus-test1 | Load GIST-SMALL-100K None | 2024022921 (task_runner.py:288)
2024-02-29 21:06:38,234 | INFO | Milvus-test1 | Performance OpenAI-LARGE-5M None | 2024022921 (task_runner.py:288)
2024-02-29 21:06:38,234 | INFO | Milvus-test1 | Performance LAION-LARGE-100M None | 2024022921 (task_runner.py:288)
2024-02-29 21:06:38,234 | INFO: task submitted: id=fe4bb641681a4899a2598c48fdee9497, 2024022921, case number: 3 (interface.py:235) (939332)
2024-02-29 21:06:39,156 | INFO: [1/3] start case: {'label': <CaseLabel.Load: 1>, 'dataset': {'data': {'name': 'GIST', 'size': 100000, 'dim': 960, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus-test1'}, drop_old=True (interface.py:167) (939903)
2024-02-29 21:07:09,514 | WARNING: pre run case error: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)> (task_runner.py:92) (939903)
2024-02-29 21:07:09,514 | WARNING: [1/3] case {'label': <CaseLabel.Load: 1>, 'dataset': {'data': {'name': 'GIST', 'size': 100000, 'dim': 960, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus-test1'} failed to run, reason=<MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)> (interface.py:187) (939903)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/interface.py", line 168, in _async_task_v2
case_res.metrics = runner.run(drop_old)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 96, in run
self._pre_run(drop_old)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 93, in _pre_run
raise e from None
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 86, in _pre_run
self.init_db(drop_old)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 77, in init_db
self.db = db_cls(
^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/clients/milvus/milvus.py", line 43, in __init__
connections.connect(**self.db_config, timeout=30)
File "/usr/local/lib/python3.11/site-packages/pymilvus/orm/connections.py", line 414, in connect
connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
File "/usr/local/lib/python3.11/site-packages/pymilvus/orm/connections.py", line 365, in connect_milvus
gh._wait_for_channel_ready(timeout=timeout)
File "/usr/local/lib/python3.11/site-packages/pymilvus/client/grpc_handler.py", line 146, in _wait_for_channel_ready
raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>@gaowayne seems that the milvus service is not being deployed correctly.
gaowayne
commented
7 months ago @gaowayne seems that the milvus service is not being deployed correctly.
thank you so much man. if I would like to make Milvus use another nvme ssd, which can I control this? I saw vectordb storage engine actually is ectd + minio. :) is there any other vectorDB that is single node single application? :)
gaowayne
commented
7 months ago @gaowayne seems that the milvus service is not being deployed correctly. FYI, https://milvus.io/docs/install_standalone-docker.md
thank you so much man. if I would like to make Milvus use another nvme ssd, which can I control this? I saw vectordb storage engine actually is ectd + minio. :) is there any other vectorDB that is single node single application? :)
I got it, it just use the drive we start milvus docker. :) it is OK. I already run this. maybe I know if I would like give NVMe SSD more pressure, what I can tune, I saw NVMe is pretty idle now.
alwayslove2013
commented
7 months ago @XuanYang-cn Could you help it?
if I would like to make Milvus use another nvme ssd, which can I control this? I saw vectordb storage engine actually is ectd + minio. :) is there any other vectorDB that is single node single application? :)
gaowayne
commented
7 months ago guys, what is reason below error, I already make milvus run well and hello example runs OK with my container setup but I still suffer this error
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-083-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-083-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-084-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-084-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-085-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-085-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-086-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-086-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-087-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-087-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-088-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-088-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-089-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-089-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-090-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-090-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-091-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-091-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-092-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-092-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-093-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-093-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-094-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-094-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-095-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-095-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-096-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-096-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-097-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-097-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,089 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-098-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-098-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,090 | INFO: local file: /mnt/nvme5n1/vectordb/dataset/laion/laion_large_100m/train-099-of-100.parquet not match with remote: assets.zilliz.com/benchmark/laion_large_100m/train-099-of-100.parquet; add to downloading list (data_source.py:136) (1166455)
2024-03-02 09:34:25,090 | INFO: Start to downloading files, total count: 102 (data_source.py:142) (1166455)
0%| | 0/102 [00:00<?, ?it/s]
2024-03-02 09:34:25,300 | WARNING: pre run case error: The specified key does not exist. (task_runner.py:92) (1166455)
2024-03-02 09:34:25,301 | WARNING: [1/1] case {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus-test6'} failed to run, reason=The specified key does not exist. (interface.py:187) (1166455)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 96, in run
self._pre_run(drop_old)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 93, in _pre_run
raise e from None
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/task_runner.py", line 87, in _pre_run
self.ca.dataset.prepare(self.dataset_source)
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/dataset.py", line 202, in prepare
source.reader().read(
File "/usr/local/lib/python3.11/site-packages/vectordb_bench/backend/data_source.py", line 145, in read
self.fs.download(s3_file, local_ds_root.as_posix())
File "/usr/local/lib/python3.11/site-packages/fsspec/spec.py", line 1534, in download
return self.get(rpath, lpath, recursive=recursive, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 650, in _get
return await _run_coros_in_chunks(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 254, in _run_coros_in_chunks
await asyncio.gather(*chunk, return_exceptions=return_exceptions),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/asyncio/tasks.py", line 452, in wait_for
return await fut
^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fsspec/callbacks.py", line 81, in func
return await fn(path1, path2, callback=child, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 1224, in _get_file
body, content_length = await _open_file(range=0)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 1215, in _open_file
resp = await self._call_s3(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 348, in _call_s3
return await _error_wrapper(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 140, in _error_wrapper
raise err
FileNotFoundError: The specified key does not exist.
2024-03-02 09:34:25,303 | INFO |Task summary: run_id=6f628, task_label=2024030209 (models.py:285)
2024-03-02 09:34:25,303 | INFO |DB | db_label case label | load_dur qps latency(p99) recall max_load_count | label (models.py:285)
2024-03-02 09:34:25,303 | INFO |------ | -------- ------------------- ---------- | ----------- ---------- --------------- ------------- -------------- | ----- (models.py:285)
2024-03-02 09:34:25,303 | INFO |Milvus | test6 Performance768D100M 2024030209 | 0.0 0.0 0.0 0.0 0 | x (models.py:285)
2024-03-02 09:34:25,303 | WARNING: Replacing existing result with the same file_name: /usr/local/lib/python3.11/site-packages/vectordb_bench/results/Milvus/result_20240302_2024030209_milvus.json (models.py:139) (1166455)
2024-03-02 09:34:25,303 | INFO: write results to disk /usr/local/lib/python3.11/site-packages/vectordb_bench/results/Milvus/result_20240302_2024030209_milvus.json (models.py:143) (1166455)
2024-03-02 09:34:25,304 | INFO: Succes to finish task: label=2024030209, run_id=6f6282a032124975a2aeae99156be307 (interface.py:207) (1166455)
XuanYang-cn
commented
7 months ago same as: #275, should be fixed by https://github.com/zilliztech/VectorDBBench/pull/287 The fixes will be released at 0.0.7 milestone
XuanYang-cn
commented
6 months ago 0.0.7 released, should be fixed
please see below error during install
as we can see I did not install setuptools