search

erikbern / ann-benchmarks

Benchmarks of approximate nearest neighbor libraries in Python
http://ann-benchmarks.com
MIT License
4.73k stars 715 forks source link

Optimize the peak memory usage when loading dataset. #514

Open Light-V opened 2 months ago

Light-V commented 2 months ago

I've made a small optimization in data loading process that reduces peak memory usage when handling large datasets. Previously, we were using np.array() to convert large datasets from h5py objects into NumPy arrays. This operation, while straightforward, was causing a spike in memory usage due to the creation of a new array copy, which could lead to OOM errors when working with particularly large datasets. To mitigate this issue, I have replaced np.array() with np.asarray() in the relevant sections of the code. The np.asarray() function, unlike np.array(), will attempt to pass through the input data without creating a new array copy if the input is already a NumPy array. This behavior helps to reduce unnecessary memory allocation and can be particularly effective in scenarios where memory is a constraint.

maumueller commented 2 months ago

Thanks @Light-V. Did you observe a change in index building time/search performance with this change? I have made very bad experiences without this cast, but this was a long time ago.

Light-V commented 2 months ago

@maumueller Thank you for your suggestion. I will give it a try and post the performance comparison later.

Light-V commented 2 months ago

Hi, @maumueller

I have run ann-benchmark on qsg-ngt with these two different ways to load dataset. And here's the search result:

diff result.1 result.2 3c3 < 0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9433.409

0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9616.273 5c5 < 1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.461 40888.565

1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.462 41178.560 7c7 < 2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.922 19599.222

2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.923 19792.332 9c9 < 3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.730 24640.807

3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.732 24812.362 11c11 < 4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.980 13130.141

4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.981 13396.878 13c13 < 5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.626 28672.532

5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.628 28876.686 15c15 < 6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7805.384

6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7890.528 17c17 < 7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.954 16309.917

7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.956 16625.485 19c19 < 8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14523.425

8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14687.466 21c21 < 9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.813 27601.332

9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.814 27835.597 23c23 < 10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 2951.642

10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 3008.100 25c25 < 11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.998 7070.455

11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.999 7199.444 27c27 < 12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.696 28557.780

12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.698 28513.283 29c29 < 13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.875 22116.188

13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.876 22458.581 31c31 < 14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.980 11920.713

14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.981 12182.230 33c33 < 15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4511.494

15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4587.789 35c35 < 16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.991 10027.577

16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.992 10132.040 37c37 < 17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.845 21594.956

17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.844 21941.602 39c39 < 18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.910 18668.998

18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.912 19092.833 41c41 < 19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.538 31914.098

19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.536 32022.208 43c43 < 20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.625 36063.251

20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.628 36227.625 45c45 < 21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.803 25712.051

21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.804 25995.405 47c47 < 22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4170.250

22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4243.505 49c49 < 23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19146.093

23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19330.184 51c51 < 24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32339.626

24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32329.880 53c53 < 25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.955 16193.192

25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.956 16318.249 55c55 < 26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.841 19298.415

26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.840 19437.719 57c57 < 27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2212.972

27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2235.706 59c59 < 28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.771 25701.573

28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.772 25698.865 61c61 < 29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.860 22676.155

29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.861 23154.836 63c63 < 30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.916 16764.549

30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.913 16869.750

And the build time is 3791.798057794571s and 3794.1561596393585s. I think there is no significant different between them.