search

weaviate / weaviate-benchmarking

Tools for various benchmarking scenarios
BSD 3-Clause "New" or "Revised" License
25 stars 13 forks source link

add parallelization for import data, log and save stop index per process #2

Closed StefanBogdan closed 6 months ago

StefanBogdan commented 2 years ago

Considering that it is late I might have missed something, I tested it a bit and I can see some stuff being done in parallel:

weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:01.775 | INFO     | functions:import_into_weaviate:290 - Start import process for sift-128-euclidean.hdf5, ef64, maxConnections64
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:22.280 | INFO     | functions:add_batch:24 - Import status => added 10000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:26.997 | INFO     | functions:add_batch:24 - Import status => added 10000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:40.884 | INFO     | functions:add_batch:24 - Import status => added 10000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:45.536 | INFO     | functions:add_batch:24 - Import status => added 20000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:50:53.654 | INFO     | functions:add_batch:24 - Import status => added 10000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:03.695 | INFO     | functions:add_batch:24 - Import status => added 20000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:10.530 | INFO     | functions:add_batch:24 - Import status => added 20000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:21.588 | INFO     | functions:add_batch:24 - Import status => added 20000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:38.210 | INFO     | functions:add_batch:24 - Import status => added 30000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:38.642 | INFO     | functions:add_batch:24 - Import status => added 30000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:51:43.958 | INFO     | functions:add_batch:24 - Import status => added 30000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:02.528 | INFO     | functions:add_batch:24 - Import status => added 30000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:09.422 | INFO     | functions:add_batch:24 - Import status => added 40000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:17.006 | INFO     | functions:add_batch:24 - Import status => added 40000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:32.766 | INFO     | functions:add_batch:24 - Import status => added 40000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:33.839 | INFO     | functions:add_batch:24 - Import status => added 50000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:52:43.216 | INFO     | functions:add_batch:24 - Import status => added 40000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:53:04.304 | INFO     | functions:add_batch:24 - Import status => added 50000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:53:04.403 | INFO     | functions:add_batch:24 - Import status => added 60000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:53:18.724 | INFO     | functions:add_batch:24 - Import status => added 50000 of 125000 objects
weaviate-benchmarking-benchmark-ann-1  | 2022-05-23 16:53:32.738 | INFO     | functions:add_batch:24 - Import status => added 50000 of 125000 objects

Let me know what do you think.

weaviate-git-bot commented 1 year ago

Great to see you again! Thanks for the contribution.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

trengrj commented 6 months ago

Closing as v2 branch has these features already