baidu / puck

Puck is a high-performance ANN search engine
Apache License 2.0
343 stars 37 forks source link

puck::PuckIndex::train: free(): invalid next size (normal) #33

Open dqxcj opened 2 days ago

dqxcj commented 2 days ago

bug表现

在测试puck过程中发现一个偶发性bug,表现如下面两张图

image

image

gdb调试发现是puck::PuckIndex::train: 795的std::unique_ptr<float[]> kmeans_train_vocab(new float[train_vocab_len]);在释放时报的错:

(gdb) until
free(): invalid next size (normal)
[Thread 0x7fff21ffb700 (LWP 965310) exited]

Thread 1 "test_puck" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe7d25300 (LWP 945491)]
0x00007fffeb0e770f in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007fffeb0e770f in raise () from /lib64/libc.so.6
#1  0x00007fffeb0d1b25 in abort () from /lib64/libc.so.6
#2  0x00007fffeb12a897 in __libc_message () from /lib64/libc.so.6
#3  0x00007fffeb130fdc in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fffeb132c1c in _int_free () from /lib64/libc.so.6
#5  0x00000000008aeddb in std::default_delete<float []>::operator()<float> (this=0x7fffffffb880, __ptr=0x395b7c0)
    at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/unique_ptr.h:115
#6  0x00000000008adbaf in std::unique_ptr<float [], std::default_delete<float []> >::~unique_ptr (this=0x7fffffffb880, 
    __in_chrg=<optimized out>) at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/unique_ptr.h:529
#7  0x00007ffff7ea7bce in puck::PuckIndex::train (this=0xfdb5a0) at external/puck/puck/puck/puck_index.cpp:795

怀疑是往kmeans_train_vocab填充数据时内存越界了,导致train结束释放kmeans_train_vocab时出现问题。

在复现时发现,召回率不理想的小数据集(80k行,32维,top300召回率10%)出现这个bug的概率较高(大数据集跑起来太慢,没咋测试)

环境信息

Hanano-Yuuki commented 2 days ago

Hi dqxcj, Thank you for your feedback. We have tested with our dataset (80k, 32d) and did not encounter this issue. If possible, could you provide the specific parameters you used during the training? Sharing the dataset you used for testing would also be helpful for us to better understand and reproduce the problem.

dqxcj commented 2 days ago

Hi dqxcj, Thank you for your feedback. We have tested with our dataset (80k, 32d) and did not encounter this issue. If possible, could you provide the specific parameters you used during the training? Sharing the dataset you used for testing would also be helpful for us to better understand and reproduce the problem.

参数如下: I1120 11:58:53.928615 206707 index_conf.cpp:204] feature_dim = 32 I1120 11:58:53.928618 206707 index_conf.cpp:205] whether_norm = 0 I1120 11:58:53.928620 206707 index_conf.cpp:206] index_type = 1 I1120 11:58:53.928629 206707 index_conf.cpp:207] total_point_count = 83790 I1120 11:58:53.928632 206707 index_conf.cpp:210] coarse_cluster_count = 500 I1120 11:58:53.928634 206707 index_conf.cpp:211] fine_cluster_count = 500 I1120 11:58:53.928638 206707 index_conf.cpp:212] search_coarse_count = 200 I1120 11:58:53.928643 206707 index_conf.cpp:221] radius_rate = 1 I1120 11:58:53.928653 206707 index_conf.cpp:223] whether_filter = 1 I1120 11:58:53.928655 206707 index_conf.cpp:224] filter_nsq = 8 I1120 11:58:53.928658 206707 index_conf.cpp:225] ks = 256 I1120 11:58:53.928668 206707 index_conf.cpp:229] whether_pq = 1 I1120 11:58:53.928671 206707 index_conf.cpp:230] nsq = 32 I1120 11:58:53.928673 206707 index_conf.cpp:231] ks = 256 I1120 11:58:53.928675 206707 index_conf.cpp:235] topk = 100 I1120 11:58:53.928678 206707 index_conf.cpp:241] /data/workspace/ann/test/data/puck/20241119/index/all_data.url

dqxcj commented 2 days ago

在puck assign也偶尔会出现core

I1119 04:14:54.293910 289901 puck_index.cpp:749] puck_assign, thread_params.start_id = 0 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 0
I1119 04:14:54.293926 289902 puck_index.cpp:749] puck_assign, thread_params.start_id = 250367 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 1
I1119 04:14:54.293988 289903 puck_index.cpp:749] puck_assign, thread_params.start_id = 500734 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 2
I1119 04:14:54.294030 289904 puck_index.cpp:749] puck_assign, thread_params.start_id = 751101 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 3
I1119 04:14:54.294107 289905 puck_index.cpp:749] puck_assign, thread_params.start_id = 1001468 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 4
I1119 04:14:54.294231 289906 puck_index.cpp:749] puck_assign, thread_params.start_id = 1251835 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 5
I1119 04:14:54.294389 289907 puck_index.cpp:749] puck_assign, thread_params.start_id = 1502202 points_count = 250367 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 6
I1119 04:14:54.294400 289908 puck_index.cpp:749] puck_assign, thread_params.start_id = 1752569 points_count = 250366 feature_file_name = /data/home/ann/test/data/puck/20241118/index/all_data.feat.bin threadId = 7
I1119 04:15:11.216840 289907 puck_index.cpp:715] puck assign from 1502202 cost 16922 ms.
double free or corruption (out)
I1120 12:13:16.599362 4004066 puck_index.cpp:749] puck_assign, thread_params.start_id = 0 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 0
I1120 12:13:16.599406 4004067 puck_index.cpp:749] puck_assign, thread_params.start_id = 41900 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 1
I1120 12:13:16.599460 4004068 puck_index.cpp:749] puck_assign, thread_params.start_id = 83800 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 2
I1120 12:13:16.599469 4004069 puck_index.cpp:749] puck_assign, thread_params.start_id = 125700 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 3
I1120 12:13:16.599599 4004070 puck_index.cpp:749] puck_assign, thread_params.start_id = 167600 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 4
I1120 12:13:16.599704 4004071 puck_index.cpp:749] puck_assign, thread_params.start_id = 209500 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 5
I1120 12:13:16.599853 4004072 puck_index.cpp:749] puck_assign, thread_params.start_id = 251400 points_count = 41900 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 6
I1120 12:13:16.599864 4004073 puck_index.cpp:749] puck_assign, thread_params.start_id = 293300 points_count = 41895 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 7
I1120 12:13:19.522874 4004068 puck_index.cpp:715] puck assign from 83800 cost 2923 ms.
double free or corruption (out)
Aborted (core dumped)
I1120 12:54:05.014528 4082265 puck_index.cpp:749] puck_assign, thread_params.start_id = 0 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 0
I1120 12:54:05.014561 4082266 puck_index.cpp:749] puck_assign, thread_params.start_id = 52377 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 1
I1120 12:54:05.014605 4082267 puck_index.cpp:749] puck_assign, thread_params.start_id = 104754 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 2
I1120 12:54:05.014714 4082268 puck_index.cpp:749] puck_assign, thread_params.start_id = 157131 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 3
I1120 12:54:05.014767 4082269 puck_index.cpp:749] puck_assign, thread_params.start_id = 209508 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 4
I1120 12:54:05.014882 4082270 puck_index.cpp:749] puck_assign, thread_params.start_id = 261885 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 5
I1120 12:54:05.015053 4082271 puck_index.cpp:749] puck_assign, thread_params.start_id = 314262 points_count = 52377 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 6
I1120 12:54:05.015065 4082272 puck_index.cpp:749] puck_assign, thread_params.start_id = 366639 points_count = 52370 feature_file_name = /data/home/ann/test/data/puck/20241119/index/all_data.feat.bin threadId = 7
I1120 12:54:08.666918 4082265 puck_index.cpp:715] puck assign from 0 cost 3652 ms.
munmap_chunk(): invalid pointer
Aborted (core dumped)
Hanano-Yuuki commented 2 days ago

Hi dqxcj,

We apologize for not being able to reproduce the issue with the parameters you provided. However, we have identified a potential memory corruption issue in the file puck/puck/quantization.cpp at line 142, where a unique_ptr is taking ownership of memory allocated by posix_memalign, which could lead to destructor errors. We are not certain if this is the cause of the problem you're experiencing.

As a result, we have pushed a new branch named fix_mem_corrupt to address this issue. Could you please check if the problem mentioned in this issue still persists? Thank you for your cooperation.

dqxcj commented 2 days ago

好的,感谢,我试试

dqxcj commented 2 days ago

还是会core:

I1120 17:05:28.135869 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 17:05:42.524760 589223 puck_index.cpp:867] deviation error of init sub 27 pq codebook clusters is 6.39904e-06
I1120 17:05:42.530800 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 17:05:56.387727 589223 puck_index.cpp:867] deviation error of init sub 28 pq codebook clusters is 5.03276e-06
I1120 17:05:56.393782 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 17:06:10.705046 589223 puck_index.cpp:867] deviation error of init sub 29 pq codebook clusters is 3.57208e-06
I1120 17:06:10.710629 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 17:06:24.643358 589223 puck_index.cpp:867] deviation error of init sub 30 pq codebook clusters is 7.86053e-06
I1120 17:06:24.648994 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 17:06:38.710780 589223 puck_index.cpp:867] deviation error of init sub 31 pq codebook clusters is 1.05451e-06
I1120 17:06:38.716567 589223 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
E1120 17:06:41.142864 589223 kmeans.cpp:303] kmeans: reassigned 1210 times, abandoning
I1120 17:06:41.142906 589223 puck_index.cpp:867] deviation error of init sub 32 pq codebook clusters is -1
I1120 17:06:41.147152 589223 puck_index.cpp:878] 0 suc.
I1120 17:06:41.147176 589223 puck_index.cpp:127] PuckIndex save coodbooks
I1120 17:06:41.147186 589223 hierarchical_cluster_index.cpp:327] HierarchicalClusterIndex start save index
I1120 17:06:41.147365 589223 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/coarse.dat
I1120 17:06:41.147578 589223 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/fine.dat
I1120 17:06:41.147912 591433 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/filter_codebook.dat
I1120 17:06:41.147969 591434 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/learn_codebooks.dat
double free or corruption (!prev)
Aborted (core dumped)
Hanano-Yuuki commented 2 days ago

Hi dqxcj,

We have noticed that your log shows: I1120 17:06:41.142906 589223 puck_index.cpp:867] deviation error of init sub 32 pq codebook clusters is -1

If the nsq parameter is set to 32 and the dataset dimension is 32, there should not be such output for reporting the sub-dimension 32 clustering error. Please provide the complete log information and executed command line, double-check the train/build parameters as well as the dataset to ensure they are valid. This will help us further diagnose the issue.

The relevant code context is located at lines 849 and 867 in the file puck/puck/puck_index.cpp.

nk2014yj commented 2 days ago
image

这次的core是kmeans 收敛不了导致的。一般情况下,kmeans的训练数据集要>= 聚类中心的20倍,且是数据集是dense embedding基本上都会收敛。

有两个办法可以试试, 1.扩大数据规模 or 缩小聚类中心个数。 puck默认coarse & fine聚类中心训练数据规模500w,pq训练规模100w,这种量级下一般都会收敛。如果数据规模太小,可以试试缩小coarse_cluster_count 和 fine_cluster_count,观察一下“kmeans: reassigned”有没有这个报错。pq的训练聚类中心是256,只能扩大训练数据规模试试。 2.上面的办法解决不了的话,大概率是这个数据集分布不太适合kmeans。即使部分时候可以聚类成功,分组效果也会比较差从而导致召回率低。对于这种数据集,推荐使用tinker,设置index_type=2,使用较小的coarse_cluster_count 和 fine_cluster_count。

dqxcj commented 2 days ago

Hi dqxcj,

We have noticed that your log shows: I1120 17:06:41.142906 589223 puck_index.cpp:867] deviation error of init sub 32 pq codebook clusters is -1

If the nsq parameter is set to 32 and the dataset dimension is 32, there should not be such output for reporting the sub-dimension 32 clustering error. Please provide the complete log information and executed command line, double-check the train/build parameters as well as the dataset to ensure they are valid. This will help us further diagnose the issue.

The relevant code context is located at lines 849 and 867 in the file puck/puck/puck_index.cpp.

明白了,这里我的nsq确实设置的不是32,而是33,谢谢。

dqxcj commented 2 days ago

image 这次的core是kmeans 收敛不了导致的。一般情况下,kmeans的训练数据集要>= 聚类中心的20倍,且是数据集是dense embedding基本上都会收敛。 有两个办法可以试试, 1.扩大数据规模 or 缩小聚类中心个数。 puck默认coarse & fine聚类中心训练数据规模500w,pq训练规模100w,这种量级下一般都会收敛。如果数据规模太小,可以试试缩小coarse_cluster_count 和 fine_cluster_count,观察一下“kmeans: reassigned”有没有这个报错。pq的训练聚类中心是256,只能扩大训练数据规模试试。 2.上面的办法解决不了的话,大概率是这个数据集分布不太适合kmeans。即使部分时候可以聚类成功,分组效果也会比较差从而导致召回率低。对于这种数据集,推荐使用tinker,设置index_type=2,使用较小的coarse_cluster_count 和 fine_cluster_count。

好的,我试试,谢谢。

dqxcj commented 2 days ago

image 这次的core是kmeans 收敛不了导致的。一般情况下,kmeans的训练数据集要>= 聚类中心的20倍,且是数据集是dense embedding基本上都会收敛。 有两个办法可以试试, 1.扩大数据规模 or 缩小聚类中心个数。 puck默认coarse & fine聚类中心训练数据规模500w,pq训练规模100w,这种量级下一般都会收敛。如果数据规模太小,可以试试缩小coarse_cluster_count 和 fine_cluster_count,观察一下“kmeans: reassigned”有没有这个报错。pq的训练聚类中心是256,只能扩大训练数据规模试试。 2.上面的办法解决不了的话,大概率是这个数据集分布不太适合kmeans。即使部分时候可以聚类成功,分组效果也会比较差从而导致召回率低。对于这种数据集,推荐使用tinker,设置index_type=2,使用较小的coarse_cluster_count 和 fine_cluster_count。

我尝试将聚类中心个缩小为100,发现还是报错,而且从偶发性变成了必发性,可能确实数据规模太小了或者这个数据集不适合kmeans,或者我的配置有其它错误吗? 配置:

feature_dim: 32
coarse_cluster_count: 100
fine_cluster_count: 100
train_points_count: 500000
pq_train_points_count: 1000000
nsq: 32
filter_nsq: 8
whether_norm: false
ip2cos: 1

日志:

I1120 20:13:48.582993 245964 hierarchical_cluster_index.cpp:203] 1
 start to train
I1120 20:13:53.552618 245964 hierarchical_cluster_index.cpp:1372] total_point_count for train is 83753
I1120 20:13:53.552642 245964 hierarchical_cluster_index.cpp:598] 4096 443420 446464
I1120 20:13:53.552663 245964 quantization.cpp:38] QuantizationParams.dim = 33, QuantizationParams.ks = 256, QuantizationParams.lsq = 5, QuantizationParams.nsq = 8
I1120 20:13:53.552672 245964 quantization.cpp:57] _per_fea_len=12
I1120 20:13:53.552675 245964 quantization.cpp:58] _per_subspace_len=1
I1120 20:13:53.552676 245964 quantization.cpp:59] _params.nsq=8
I1120 20:13:53.552685 245964 quantization.cpp:38] QuantizationParams.dim = 33, QuantizationParams.ks = 256, QuantizationParams.lsq = 1, QuantizationParams.nsq = 33
I1120 20:13:53.552693 245964 quantization.cpp:57] _per_fea_len=37
I1120 20:13:53.552695 245964 quantization.cpp:58] _per_subspace_len=1
I1120 20:13:53.552697 245964 quantization.cpp:59] _params.nsq=33
I1120 20:13:53.758903 245964 hierarchical_cluster_index.cpp:1401] cur_index_path = mid-data
I1120 20:13:53.758948 245964 hierarchical_cluster_index.cpp:110] start write_fvec_format mid-data/train_clusters.dat
I1120 20:13:53.779044 245964 hierarchical_cluster_index.cpp:883] HierarchicalClusterIndex start train
I1120 20:13:53.783592 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:03.169279 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 8.0828 when ite = 0
I1120 20:14:03.173281 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:12.441591 245964 hierarchical_cluster_index.cpp:925] 0 deviation error of init fine clusters is 4.99478 when ite = 0
I1120 20:14:12.445679 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:21.709494 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.33484 when ite = 1
I1120 20:14:21.713416 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:30.982048 245964 hierarchical_cluster_index.cpp:925] 1 deviation error of init fine clusters is 5.08264 when ite = 1
I1120 20:14:30.982097 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.08264 / 4.99478, params.niter = 30
I1120 20:14:30.985921 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:43.323585 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.31972 when ite = 2
I1120 20:14:43.327664 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:14:56.000748 245964 hierarchical_cluster_index.cpp:925] 2 deviation error of init fine clusters is 5.07097 when ite = 2
I1120 20:14:56.000797 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.07097 / 4.99478, params.niter = 40
I1120 20:14:56.004740 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:15:11.726043 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.28355 when ite = 3
I1120 20:15:11.730340 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:15:27.343252 245964 hierarchical_cluster_index.cpp:925] 3 deviation error of init fine clusters is 5.05633 when ite = 3
I1120 20:15:27.343308 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.05633 / 4.99478, params.niter = 50
I1120 20:15:27.347282 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:15:38.780994 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.19554 when ite = 4
I1120 20:15:38.785456 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:15:57.560093 245964 hierarchical_cluster_index.cpp:925] 4 deviation error of init fine clusters is 5.01245 when ite = 4
I1120 20:15:57.560140 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.01245 / 4.99478, params.niter = 60
I1120 20:15:57.564121 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:16:14.938707 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.31701 when ite = 5
I1120 20:16:14.942533 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:16:36.608397 245964 hierarchical_cluster_index.cpp:925] 5 deviation error of init fine clusters is 5.05082 when ite = 5
I1120 20:16:36.608445 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.05082 / 4.99478, params.niter = 70
I1120 20:16:36.612360 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:17:00.010006 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.30688 when ite = 6
I1120 20:17:00.013886 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:17:24.877298 245964 hierarchical_cluster_index.cpp:925] 6 deviation error of init fine clusters is 5.05069 when ite = 6
I1120 20:17:24.877346 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.05069 / 4.99478, params.niter = 80
I1120 20:17:24.881409 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:17:38.797832 245964 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.34244 when ite = 7
I1120 20:17:38.801936 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 20:18:07.360877 245964 hierarchical_cluster_index.cpp:925] 7 deviation error of init fine clusters is 5.08923 when ite = 7
I1120 20:18:07.360932 245964 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.08923 / 4.99478, params.niter = 90
I1120 20:18:07.360949 245964 hierarchical_cluster_index.cpp:983] init coarse & fine clusters and alpha vocab cost 253581 ms
I1120 20:18:07.361714 245964 hierarchical_cluster_index.cpp:327] HierarchicalClusterIndex start save index
I1120 20:18:07.361847 245964 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/coarse.dat
I1120 20:18:07.361887 245964 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/workspace/ann/test/data/puck/20241120/index/fine.dat
I1120 20:18:07.362783 245964 hierarchical_cluster_index.cpp:258] start load index file /data/workspace/ann/test/data/puck/20241120/index/coarse.dat
I1120 20:18:07.362818 245964 hierarchical_cluster_index.cpp:283] start load index file /data/workspace/ann/test/data/puck/20241120/index/fine.dat
I1120 20:18:07.363137 245964 hierarchical_cluster_index.cpp:322] HierarchicalClusterIndex::read_coodbooks Suc.
I1120 20:18:07.547756 245964 puck_index.cpp:804] true point cnt for puck train = 83753
I1120 20:18:07.547791 245964 hierarchical_cluster_index.cpp:110] start write_fvec_format mid-data/train_pq.dat
I1120 20:18:07.565013 245964 hierarchical_cluster_index.cpp:1178] HierarchicalClusterIndex::batch_assign
I1120 20:18:07.565240 246477 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 0 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 0
I1120 20:18:07.565517 246477 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.565521 246479 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 3490 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 2
I1120 20:18:07.565615 246478 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 1745 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 1
I1120 20:18:07.565718 246479 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.565733 246480 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 5235 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 3
I1120 20:18:07.565763 246478 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.565788 246480 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566125 246481 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 6980 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 4
I1120 20:18:07.566136 246482 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 8725 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 5
I1120 20:18:07.566212 246481 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566216 246482 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566298 246483 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 10470 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 6
I1120 20:18:07.566424 246483 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566459 246484 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 12215 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 7
I1120 20:18:07.566504 246484 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566576 246485 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 13960 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 8
I1120 20:18:07.566613 246485 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.566676 246486 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 15705 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 9
I1120 20:18:07.566761 246487 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 17450 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 10
I1120 20:18:07.566828 246487 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.567108 246486 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.567140 246488 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 19195 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 11
I1120 20:18:07.567262 246489 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 20940 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 12
I1120 20:18:07.567319 246489 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.567348 246488 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.567662 246490 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 22685 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 13
I1120 20:18:07.567754 246491 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 24430 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 14
I1120 20:18:07.568109 246490 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.568148 246491 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.568150 246492 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 26175 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 15
I1120 20:18:07.568212 246492 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.568553 246493 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 27920 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 16
I1120 20:18:07.568642 246493 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.568811 246494 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 29665 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 17
I1120 20:18:07.568948 246494 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.569187 246495 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 31410 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 18
I1120 20:18:07.569252 246495 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.569577 246496 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 33155 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 19
I1120 20:18:07.570205 246497 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 34900 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 20
I1120 20:18:07.570523 246498 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 36645 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 21
I1120 20:18:07.570976 246496 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.571149 246498 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.571193 246499 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 38390 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 22
I1120 20:18:07.571286 246499 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.571450 246500 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 40135 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 23
I1120 20:18:07.571538 246500 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.571713 246501 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 41880 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 24
I1120 20:18:07.571800 246501 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.571980 246502 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 43625 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 25
I1120 20:18:07.572222 246497 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.572441 246502 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.572682 246503 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 45370 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 26
I1120 20:18:07.573997 246503 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.574384 246504 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 47115 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 27
I1120 20:18:07.574443 246504 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.574852 246506 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 50605 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 29
I1120 20:18:07.574872 246507 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 52350 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 30
I1120 20:18:07.574896 246506 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.574918 246507 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.575601 246505 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 48860 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 28
I1120 20:18:07.576025 246509 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 55840 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 32
I1120 20:18:07.576200 246508 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 54095 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 31
I1120 20:18:07.577283 246508 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.577421 246505 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.577524 246510 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 57585 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 33
I1120 20:18:07.578001 246509 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.578058 246510 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.578289 246511 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 59330 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 34
I1120 20:18:07.580281 246511 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.580502 246512 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 61075 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 35
I1120 20:18:07.580775 246514 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 64565 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 37
I1120 20:18:07.580969 246514 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.581290 246513 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 62820 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 36
I1120 20:18:07.582271 246512 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.582864 246515 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 66310 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 38
I1120 20:18:07.583257 246513 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.583407 246516 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 68055 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 39
I1120 20:18:07.584306 246515 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.584386 246517 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 69800 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 40
I1120 20:18:07.585350 246516 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.585533 246518 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 71545 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 41
I1120 20:18:07.586792 246518 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.586858 246519 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 73290 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 42
I1120 20:18:07.586967 246517 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.596158 246519 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.610021 246522 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 78525 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 45
I1120 20:18:07.610086 246522 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.610541 246521 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 76780 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 44
I1120 20:18:07.610596 246521 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.611137 246524 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 82015 points_count = 1738 feature_file_name = mid-data/train_pq.dat threadId = 47
I1120 20:18:07.611162 246524 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.612150 246523 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 80270 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 46
I1120 20:18:07.612175 246523 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.612598 246520 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 75035 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 43
I1120 20:18:07.612640 246520 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 20:18:07.646950 245964 hierarchical_cluster_index.cpp:1245] batch_assign succeeded, deviation error = 2.44756, pruning computation = 0.922135
I1120 20:18:07.652354 245964 puck_index.cpp:832] 8 5 256 5
I1120 20:18:07.652382 245964 puck_index.cpp:837] 33 1 256 1
I1120 20:18:07.652387 245964 puck_index.cpp:846] 8 5
I1120 20:18:07.653295 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:18:24.277140 245964 puck_index.cpp:867] deviation error of init sub 0 pq codebook clusters is 0.131303
I1120 20:18:24.280230 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:18:40.858407 245964 puck_index.cpp:867] deviation error of init sub 1 pq codebook clusters is 0.118229
I1120 20:18:40.861636 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:18:57.574046 245964 puck_index.cpp:867] deviation error of init sub 2 pq codebook clusters is 0.132433
I1120 20:18:57.577081 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:19:14.290277 245964 puck_index.cpp:867] deviation error of init sub 3 pq codebook clusters is 0.140133
I1120 20:19:14.293483 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:19:31.174304 245964 puck_index.cpp:867] deviation error of init sub 4 pq codebook clusters is 0.118466
I1120 20:19:31.177736 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:19:47.957939 245964 puck_index.cpp:867] deviation error of init sub 5 pq codebook clusters is 0.124241
I1120 20:19:47.961313 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:20:04.734012 245964 puck_index.cpp:867] deviation error of init sub 6 pq codebook clusters is 0.00301769
I1120 20:20:04.737012 245964 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 20:20:21.471698 245964 puck_index.cpp:867] deviation error of init sub 7 pq codebook clusters is 0.0222823
I1120 20:20:21.474130 245964 puck_index.cpp:878] 0 suc.
corrupted double-linked list
Aborted (core dumped)

数据格式:参考的ann-benchmarks. num + dimension + data float-80k-bin.zip

dqxcj commented 2 days ago

Hi dqxcj,

We have noticed that your log shows: I1120 17:06:41.142906 589223 puck_index.cpp:867] deviation error of init sub 32 pq codebook clusters is -1

If the nsq parameter is set to 32 and the dataset dimension is 32, there should not be such output for reporting the sub-dimension 32 clustering error. Please provide the complete log information and executed command line, double-check the train/build parameters as well as the dataset to ensure they are valid. This will help us further diagnose the issue.

The relevant code context is located at lines 849 and 867 in the file puck/puck/puck_index.cpp.

nsq设置为32仍会报错(index_conf.show()中nsq是33是因为ip2cos=1)

I1120 21:16:39.962391 1395284 hierarchical_cluster_index.cpp:203] 1
 start to train
I1120 21:16:54.148236 1395284 index_conf.cpp:204] feature_dim = 33
I1120 21:16:54.148262 1395284 index_conf.cpp:205] whether_norm = 0
I1120 21:16:54.148265 1395284 index_conf.cpp:206] index_type = 1
I1120 21:16:54.148268 1395284 index_conf.cpp:207] total_point_count = 0
I1120 21:16:54.148272 1395284 index_conf.cpp:210] coarse_cluster_count = 100
I1120 21:16:54.148275 1395284 index_conf.cpp:211] fine_cluster_count = 100
I1120 21:16:54.148278 1395284 index_conf.cpp:212] search_coarse_count = 100
I1120 21:16:54.148281 1395284 index_conf.cpp:221] radius_rate = 1
I1120 21:16:54.148301 1395284 index_conf.cpp:223] whether_filter = 0
I1120 21:16:54.148304 1395284 index_conf.cpp:224] filter_nsq = 8
I1120 21:16:54.148314 1395284 index_conf.cpp:225] ks = 256
I1120 21:16:54.148317 1395284 index_conf.cpp:229] whether_pq = 1
I1120 21:16:54.148320 1395284 index_conf.cpp:230] nsq = 33
I1120 21:16:54.148322 1395284 index_conf.cpp:231] ks = 256
I1120 21:16:54.148325 1395284 index_conf.cpp:235] topk = 100
I1120 21:16:54.148327 1395284 index_conf.cpp:238] ip2cos = 1
I1120 21:16:54.148330 1395284 index_conf.cpp:241] /data/home/puck/ann/test/data/puck/20241120/index/all_data.url
I1120 21:16:54.172793 1395284 hierarchical_cluster_index.cpp:1372] total_point_count for train is 83753
I1120 21:16:54.172833 1395284 hierarchical_cluster_index.cpp:598] 4096 443420 446464
I1120 21:16:54.172883 1395284 quantization.cpp:38] QuantizationParams.dim = 33, QuantizationParams.ks = 256, QuantizationParams.lsq = 5, QuantizationParams.nsq = 8
I1120 21:16:54.172900 1395284 quantization.cpp:56] _per_fea_len=12
I1120 21:16:54.172905 1395284 quantization.cpp:57] _per_subspace_len=1
I1120 21:16:54.172909 1395284 quantization.cpp:58] _params.nsq=8
I1120 21:16:54.172930 1395284 quantization.cpp:38] QuantizationParams.dim = 33, QuantizationParams.ks = 256, QuantizationParams.lsq = 1, QuantizationParams.nsq = 33
I1120 21:16:54.172933 1395284 quantization.cpp:56] _per_fea_len=37
I1120 21:16:54.172936 1395284 quantization.cpp:57] _per_subspace_len=1
I1120 21:16:54.172940 1395284 quantization.cpp:58] _params.nsq=33
I1120 21:16:54.623010 1395284 hierarchical_cluster_index.cpp:1401] cur_index_path = mid-data
I1120 21:16:54.623059 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format mid-data/train_clusters.dat
I1120 21:16:54.645714 1395284 hierarchical_cluster_index.cpp:883] HierarchicalClusterIndex start train
I1120 21:16:54.649552 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:17:06.065629 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 8.01509 when ite = 0
I1120 21:17:06.072103 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:17:17.478480 1395284 hierarchical_cluster_index.cpp:925] 0 deviation error of init fine clusters is 4.97603 when ite = 0
I1120 21:17:17.484262 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:17:28.803036 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.32079 when ite = 1
I1120 21:17:28.809700 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:17:40.337020 1395284 hierarchical_cluster_index.cpp:925] 1 deviation error of init fine clusters is 5.05941 when ite = 1
I1120 21:17:40.337067 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.05941 / 4.97603, params.niter = 30
I1120 21:17:40.342716 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:17:55.435019 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.34284 when ite = 2
I1120 21:17:55.441532 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:18:10.487214 1395284 hierarchical_cluster_index.cpp:925] 2 deviation error of init fine clusters is 5.05558 when ite = 2
I1120 21:18:10.487274 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.05558 / 4.97603, params.niter = 40
I1120 21:18:10.493377 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:18:29.258373 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.35478 when ite = 3
I1120 21:18:29.264895 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:18:48.186916 1395284 hierarchical_cluster_index.cpp:925] 3 deviation error of init fine clusters is 5.03531 when ite = 3
I1120 21:18:48.186967 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.03531 / 4.97603, params.niter = 50
I1120 21:18:48.192723 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:19:10.609123 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.21989 when ite = 4
I1120 21:19:10.615576 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:19:33.225881 1395284 hierarchical_cluster_index.cpp:925] 4 deviation error of init fine clusters is 4.99573 when ite = 4
I1120 21:19:33.225932 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 4.99573 / 4.97603, params.niter = 60
I1120 21:19:33.231515 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:19:47.443495 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.55143 when ite = 5
I1120 21:19:47.449864 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:20:13.792847 1395284 hierarchical_cluster_index.cpp:925] 5 deviation error of init fine clusters is 5.06202 when ite = 5
I1120 21:20:13.792899 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.06202 / 4.97603, params.niter = 70
I1120 21:20:13.798754 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:20:30.636795 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.24642 when ite = 6
I1120 21:20:30.643182 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:21:00.448741 1395284 hierarchical_cluster_index.cpp:925] 6 deviation error of init fine clusters is 5.00673 when ite = 6
I1120 21:21:00.448791 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.00673 / 4.97603, params.niter = 80
I1120 21:21:00.454429 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:21:21.507208 1395284 hierarchical_cluster_index.cpp:909] deviation error of init coarse clusters is 5.27529 when ite = 7
I1120 21:21:21.513926 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 800
I1120 21:21:55.331476 1395284 hierarchical_cluster_index.cpp:925] 7 deviation error of init fine clusters is 5.03088 when ite = 7
I1120 21:21:55.331522 1395284 hierarchical_cluster_index.cpp:936] current deviation error > min deviation error : 5.03088 / 4.97603, params.niter = 90
I1120 21:21:55.331559 1395284 hierarchical_cluster_index.cpp:983] init coarse & fine clusters and alpha vocab cost 300685 ms
I1120 21:21:55.332147 1395284 hierarchical_cluster_index.cpp:327] HierarchicalClusterIndex start save index
I1120 21:21:55.332291 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/coarse.dat
I1120 21:21:55.332345 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/fine.dat
I1120 21:21:55.333158 1395284 hierarchical_cluster_index.cpp:258] start load index file /data/home/puck/ann/test/data/puck/20241120/index/coarse.dat
I1120 21:21:55.333194 1395284 hierarchical_cluster_index.cpp:283] start load index file /data/home/puck/ann/test/data/puck/20241120/index/fine.dat
I1120 21:21:55.333608 1395284 hierarchical_cluster_index.cpp:322] HierarchicalClusterIndex::read_coodbooks Suc.
I1120 21:21:55.771775 1395284 puck_index.cpp:805] true point cnt for puck train = 83753
I1120 21:21:55.771812 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format mid-data/train_pq.dat
I1120 21:21:55.798378 1395284 hierarchical_cluster_index.cpp:1178] HierarchicalClusterIndex::batch_assign
I1120 21:21:55.798652 1398009 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 0 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 0
I1120 21:21:55.798789 1398009 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.798875 1398010 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 1745 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 1
I1120 21:21:55.798903 1398011 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 3490 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 2
I1120 21:21:55.798935 1398010 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.798959 1398011 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799117 1398012 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 5235 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 3
I1120 21:21:55.799178 1398012 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799191 1398013 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 6980 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 4
I1120 21:21:55.799273 1398013 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799377 1398015 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 10470 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 6
I1120 21:21:55.799443 1398015 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799508 1398016 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 12215 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 7
I1120 21:21:55.799559 1398014 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 8725 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 5
I1120 21:21:55.799585 1398016 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799621 1398014 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799703 1398017 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 13960 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 8
I1120 21:21:55.799754 1398017 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799798 1398018 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 15705 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 9
I1120 21:21:55.799832 1398018 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.799942 1398019 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 17450 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 10
I1120 21:21:55.799996 1398019 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800012 1398020 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 19195 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 11
I1120 21:21:55.800051 1398020 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800129 1398021 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 20940 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 12
I1120 21:21:55.800231 1398021 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800240 1398022 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 22685 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 13
I1120 21:21:55.800272 1398022 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800503 1398023 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 24430 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 14
I1120 21:21:55.800534 1398024 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 26175 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 15
I1120 21:21:55.800544 1398023 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800631 1398024 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.800838 1398025 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 27920 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 16
I1120 21:21:55.800901 1398025 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.801085 1398027 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 31410 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 18
I1120 21:21:55.801136 1398027 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.801263 1398028 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 33155 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 19
I1120 21:21:55.801359 1398028 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.801390 1398029 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 34900 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 20
I1120 21:21:55.801486 1398029 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.801623 1398030 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 36645 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 21
I1120 21:21:55.801698 1398030 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.801836 1398031 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 38390 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 22
I1120 21:21:55.801901 1398031 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.802011 1398032 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 40135 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 23
I1120 21:21:55.802098 1398032 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.802217 1398033 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 41880 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 24
I1120 21:21:55.802304 1398033 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.802448 1398034 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 43625 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 25
I1120 21:21:55.802521 1398034 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.802640 1398035 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 45370 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 26
I1120 21:21:55.802712 1398035 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.802836 1398036 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 47115 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 27
I1120 21:21:55.802915 1398036 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.803052 1398037 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 48860 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 28
I1120 21:21:55.803138 1398037 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.803249 1398038 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 50605 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 29
I1120 21:21:55.803320 1398038 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.803493 1398039 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 52350 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 30
I1120 21:21:55.803573 1398039 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.803680 1398040 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 54095 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 31
I1120 21:21:55.803870 1398026 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 29665 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 17
I1120 21:21:55.804039 1398040 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.804198 1398041 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 55840 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 32
I1120 21:21:55.804222 1398042 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 57585 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 33
I1120 21:21:55.804239 1398041 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.804275 1398042 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.804905 1398026 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.805148 1398044 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 61075 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 35
I1120 21:21:55.805166 1398043 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 59330 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 34
I1120 21:21:55.805267 1398043 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.805298 1398045 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 62820 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 36
I1120 21:21:55.806013 1398044 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.806015 1398045 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.806104 1398046 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 64565 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 37
I1120 21:21:55.806823 1398046 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.807101 1398047 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 66310 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 38
I1120 21:21:55.807137 1398047 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.807523 1398048 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 68055 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 39
I1120 21:21:55.808172 1398050 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 71545 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 41
I1120 21:21:55.808209 1398049 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 69800 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 40
I1120 21:21:55.808470 1398048 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.809304 1398051 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 73290 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 42
I1120 21:21:55.809633 1398049 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.809787 1398051 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.809823 1398052 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 75035 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 43
I1120 21:21:55.810024 1398050 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.810606 1398053 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 76780 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 44
I1120 21:21:55.810962 1398053 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.811082 1398052 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.811290 1398054 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 78525 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 45
I1120 21:21:55.812984 1398054 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.813162 1398056 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 82015 points_count = 1738 feature_file_name = mid-data/train_pq.dat threadId = 47
I1120 21:21:55.813194 1398056 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.813428 1398055 hierarchical_cluster_index.cpp:1215] assign, thread_params.start_id = 80270 points_count = 1745 feature_file_name = mid-data/train_pq.dat threadId = 46
I1120 21:21:55.814319 1398055 hierarchical_cluster_index.cpp:1099] HierarchicalClusterIndex::assign nearest_cell_assign_batch processing 0/1
I1120 21:21:55.960178 1395284 hierarchical_cluster_index.cpp:1245] batch_assign succeeded, deviation error = 2.43226, pruning computation = 0.922593
I1120 21:21:55.970018 1395284 puck_index.cpp:833] 8 5 256 5
I1120 21:21:55.970049 1395284 puck_index.cpp:838] 33 1 256 1
I1120 21:21:55.970055 1395284 puck_index.cpp:847] 8 5
I1120 21:21:55.972313 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:22:17.474429 1395284 puck_index.cpp:868] deviation error of init sub 0 pq codebook clusters is 0.130678
I1120 21:22:17.481118 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:22:38.930425 1395284 puck_index.cpp:868] deviation error of init sub 1 pq codebook clusters is 0.118034
I1120 21:22:38.937106 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:23:00.423054 1395284 puck_index.cpp:868] deviation error of init sub 2 pq codebook clusters is 0.131823
I1120 21:23:00.429738 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:23:22.001057 1395284 puck_index.cpp:868] deviation error of init sub 3 pq codebook clusters is 0.139211
I1120 21:23:22.008076 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:23:43.560473 1395284 puck_index.cpp:868] deviation error of init sub 4 pq codebook clusters is 0.118115
I1120 21:23:43.567039 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:24:05.070489 1395284 puck_index.cpp:868] deviation error of init sub 5 pq codebook clusters is 0.125233
I1120 21:24:05.077225 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:24:26.640897 1395284 puck_index.cpp:868] deviation error of init sub 6 pq codebook clusters is 0.00301491
I1120 21:24:26.647318 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:24:48.267957 1395284 puck_index.cpp:868] deviation error of init sub 7 pq codebook clusters is 0.0222174
I1120 21:24:48.272322 1395284 puck_index.cpp:879] 0 suc.
I1120 21:24:48.272347 1395284 puck_index.cpp:847] 33 1
I1120 21:24:48.274406 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:25:07.723282 1395284 puck_index.cpp:868] deviation error of init sub 0 pq codebook clusters is 5.47743e-06
I1120 21:25:07.729629 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:25:27.165558 1395284 puck_index.cpp:868] deviation error of init sub 1 pq codebook clusters is 5.70181e-06
I1120 21:25:27.171845 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:25:46.614609 1395284 puck_index.cpp:868] deviation error of init sub 2 pq codebook clusters is 1.3164e-06
I1120 21:25:46.620906 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:26:06.081655 1395284 puck_index.cpp:868] deviation error of init sub 3 pq codebook clusters is 2.55924e-06
I1120 21:26:06.087972 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:26:25.560809 1395284 puck_index.cpp:868] deviation error of init sub 4 pq codebook clusters is 1.38771e-06
I1120 21:26:25.567404 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:26:45.077556 1395284 puck_index.cpp:868] deviation error of init sub 5 pq codebook clusters is 1.66418e-06
I1120 21:26:45.083863 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:27:04.531400 1395284 puck_index.cpp:868] deviation error of init sub 6 pq codebook clusters is 3.00561e-06
I1120 21:27:04.537925 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:27:24.013079 1395284 puck_index.cpp:868] deviation error of init sub 7 pq codebook clusters is 6.07122e-06
I1120 21:27:24.019447 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:27:43.498359 1395284 puck_index.cpp:868] deviation error of init sub 8 pq codebook clusters is 6.34264e-06
I1120 21:27:43.504781 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:28:02.965793 1395284 puck_index.cpp:868] deviation error of init sub 9 pq codebook clusters is 5.05243e-06
I1120 21:28:02.972187 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:28:22.434432 1395284 puck_index.cpp:868] deviation error of init sub 10 pq codebook clusters is 5.31566e-06
I1120 21:28:22.441866 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:28:41.890602 1395284 puck_index.cpp:868] deviation error of init sub 11 pq codebook clusters is 8.58702e-06
I1120 21:28:41.896991 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:29:01.347394 1395284 puck_index.cpp:868] deviation error of init sub 12 pq codebook clusters is 7.06539e-06
I1120 21:29:01.353690 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:29:20.802338 1395284 puck_index.cpp:868] deviation error of init sub 13 pq codebook clusters is 1.40988e-05
I1120 21:29:20.808961 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:29:40.297031 1395284 puck_index.cpp:868] deviation error of init sub 14 pq codebook clusters is 6.86888e-06
I1120 21:29:40.303701 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:29:59.750460 1395284 puck_index.cpp:868] deviation error of init sub 15 pq codebook clusters is 4.57277e-06
I1120 21:29:59.756983 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:30:19.253473 1395284 puck_index.cpp:868] deviation error of init sub 16 pq codebook clusters is 1.13849e-05
I1120 21:30:19.259624 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:30:38.722235 1395284 puck_index.cpp:868] deviation error of init sub 17 pq codebook clusters is 8.13347e-06
I1120 21:30:38.728385 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:30:58.221913 1395284 puck_index.cpp:868] deviation error of init sub 18 pq codebook clusters is 5.66012e-06
I1120 21:30:58.228346 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:31:17.722115 1395284 puck_index.cpp:868] deviation error of init sub 19 pq codebook clusters is 6.5537e-06
I1120 21:31:17.728478 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:31:37.202998 1395284 puck_index.cpp:868] deviation error of init sub 20 pq codebook clusters is 2.73757e-05
I1120 21:31:37.209411 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:31:56.718147 1395284 puck_index.cpp:868] deviation error of init sub 21 pq codebook clusters is 4.34124e-06
I1120 21:31:56.724577 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:32:16.229899 1395284 puck_index.cpp:868] deviation error of init sub 22 pq codebook clusters is 7.80677e-06
I1120 21:32:16.236162 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:32:35.695137 1395284 puck_index.cpp:868] deviation error of init sub 23 pq codebook clusters is 9.21654e-06
I1120 21:32:35.701558 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:32:55.169744 1395284 puck_index.cpp:868] deviation error of init sub 24 pq codebook clusters is 7.85038e-06
I1120 21:32:55.176198 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:33:14.636086 1395284 puck_index.cpp:868] deviation error of init sub 25 pq codebook clusters is 1.20928e-05
I1120 21:33:14.642274 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:33:34.141265 1395284 puck_index.cpp:868] deviation error of init sub 26 pq codebook clusters is 6.06375e-06
I1120 21:33:34.147583 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:33:53.630189 1395284 puck_index.cpp:868] deviation error of init sub 27 pq codebook clusters is 6.69839e-06
I1120 21:33:53.636672 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:34:13.134539 1395284 puck_index.cpp:868] deviation error of init sub 28 pq codebook clusters is 4.15988e-06
I1120 21:34:13.140872 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:34:32.601744 1395284 puck_index.cpp:868] deviation error of init sub 29 pq codebook clusters is 1.00582e-05
I1120 21:34:32.607952 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:34:52.087459 1395284 puck_index.cpp:868] deviation error of init sub 30 pq codebook clusters is 1.93819e-06
I1120 21:34:52.093652 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
I1120 21:35:11.586118 1395284 puck_index.cpp:868] deviation error of init sub 31 pq codebook clusters is 7.84087e-07
I1120 21:35:11.592404 1395284 kmeans.cpp:182] true nsubset for  KMEANS_PLUS_PLUS = 2048
E1120 21:35:14.806133 1395284 kmeans.cpp:303] kmeans: reassigned 1210 times, abandoning
I1120 21:35:14.806174 1395284 puck_index.cpp:868] deviation error of init sub 32 pq codebook clusters is -1
I1120 21:35:14.810411 1395284 puck_index.cpp:879] 0 suc.
I1120 21:35:14.810441 1395284 puck_index.cpp:127] PuckIndex save coodbooks
I1120 21:35:14.810444 1395284 hierarchical_cluster_index.cpp:327] HierarchicalClusterIndex start save index
I1120 21:35:14.810621 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/coarse.dat
I1120 21:35:14.810695 1395284 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/fine.dat
I1120 21:35:14.810899 1401944 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/filter_codebook.dat
I1120 21:35:14.810905 1401945 hierarchical_cluster_index.cpp:110] start write_fvec_format /data/home/puck/ann/test/data/puck/20241120/index/learn_codebooks.dat
double free or corruption (out)
Aborted (core dumped)
nk2014yj commented 1 day ago

puck支持3种距离计算,分别配置如下: L2 : whether_norm: false COS : 默认 IP : ip2cos (部分数据集支持)

你的配置,看着是期望ip距离。ip2cos,对数据集做了一个转换,用到的是
https://dl.acm.org/doi/pdf/10.5555/3327345.3327382 里的公式(4)。

这个转换在训练之前要做,转换后的数据默认存储在puck_index/all_data.feat.bin,这个时候文件的长度得 = 样本总数 (sizeof(float) (32 + 1)维度 + sizeof(int)) 。 转换的处理可以参考 https://github.com/baidu/puck/blob/0f03b409b259b3791a8c119bfbd769ef73e3e93f/ann-benchmarks/benchmark/algorithms/puck_inmem.py#L86

这个数据集的向量模不满足需求(<1.0),所以没办法直接计算ip距离。只能用cos距离作为近似距离计算,我测试cos是ok的,可以试试修改下参数(设置whether_norm: true,去掉ip2cos: 1) 。

我试着把数据集先按比例缩小,再用ip2cos。filter_nsq = 8 会出core,filter_nsq=6是ok的。应该是一些边界条件没做检查,需要定位一下问题&修复一下bug。

dqxcj commented 1 day ago

puck支持3种距离计算,分别配置如下: L2 : whether_norm: false COS : 默认 IP : ip2cos (部分数据集支持)

你的配置,看着是期望ip距离。ip2cos,对数据集做了一个转换,用到的是 https://dl.acm.org/doi/pdf/10.5555/3327345.3327382 里的公式(4)。

这个转换在训练之前要做,转换后的数据默认存储在puck_index/all_data.feat.bin,这个时候文件的长度得 = 样本总数 (sizeof(float) (32 + 1)维度 + sizeof(int)) 。 转换的处理可以参考

https://github.com/baidu/puck/blob/0f03b409b259b3791a8c119bfbd769ef73e3e93f/ann-benchmarks/benchmark/algorithms/puck_inmem.py#L86

这个数据集的向量模不满足需求(<1.0),所以没办法直接计算ip距离。只能用cos距离作为近似距离计算,我测试cos是ok的,可以试试修改下参数(设置whether_norm: true,去掉ip2cos: 1) 。

我试着把数据集先按比例缩小,再用ip2cos。filter_nsq = 8 会出core,filter_nsq=6是ok的。应该是一些边界条件没做检查,需要定位一下问题&修复一下bug。

这里我用的确实有点迷糊,有些问题想请教一下:

  1. 使用IP距离时,如果提前将数据进行了L2归一化处理,那puck是否应该设置为L2距离?
  2. 使用IP距离时,如果提前将数据按照https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/XboxInnerProduct.pdf 中的Theorem 1进行处理,那puck应该如何设置距离度量方式?代码实现如下:
    static int IP2L2Convert(const std::vector<float>& org, std::vector<float>& convert, const ConvertParam& param) {
    convert = org;
    if (param.query_flag) {  // query向量处理
    convert.emplace_back(0.0);
    } else {  // item向量处理
    convert.emplace_back(sqrt(param.max_phi_square - Square(org)));
    }
    return 0;
    }
nk2014yj commented 1 day ago
  1. 使用IP距离时,如果提前将数据进行了L2归一化处理,那puck是否应该设置为L2距离? 如果向量是归一的,ip距离和cos距离是等价的,距离值是一样的。 这个时候,设置whether_norm = true,默认也是true。不需要额外设置其他的。返回的距离是2-2cos。
nk2014yj commented 1 day ago

2. 使用IP距离时,如果提前将数据按照https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/XboxInnerProduct.pdf 中的Theorem 1进行处理,那puck应该如何设置距离度量方式?代码实现如下:

提前做转换,对于puck来说,他处理的是转换后的32+1维的数据集,不知道32维数据集的存在。 这个数据集上要用ip2cos,还是要满足32+1维特征的模<1.0,在这个数据集上是不满足的,所以没办法用ip距离,建议用cos距离作为近似距离返回。 假如说,Theorem 1转换后,满足32+1维特征的模<1.0,需要设置ip2cos=1,返回距离2-2ip。

32维数据集的暴力搜索结果与32+1暴力结果有diff,是Theorem 1导致的。puck的理想召回结果是32+1的暴力。

dqxcj commented 1 day ago

好嘞,我消化一下,非常感谢两位大佬的帮忙

Hanano-Yuuki commented 1 day ago

puck支持3种距离计算,分别配置如下: L2 : whether_norm: false COS : 默认 IP : ip2cos (部分数据集支持)

你的配置,看着是期望ip距离。ip2cos,对数据集做了一个转换,用到的是 https://dl.acm.org/doi/pdf/10.5555/3327345.3327382 里的公式(4)。

这个转换在训练之前要做,转换后的数据默认存储在puck_index/all_data.feat.bin,这个时候文件的长度得 = 样本总数 (sizeof(float) (32 + 1)维度 + sizeof(int)) 。 转换的处理可以参考

https://github.com/baidu/puck/blob/0f03b409b259b3791a8c119bfbd769ef73e3e93f/ann-benchmarks/benchmark/algorithms/puck_inmem.py#L86

这个数据集的向量模不满足需求(<1.0),所以没办法直接计算ip距离。只能用cos距离作为近似距离计算,我测试cos是ok的,可以试试修改下参数(设置whether_norm: true,去掉ip2cos: 1) 。

我试着把数据集先按比例缩小,再用ip2cos。filter_nsq = 8 会出core,filter_nsq=6是ok的。应该是一些边界条件没做检查,需要定位一下问题&修复一下bug。

已修,推在分支fix_mem_corrupt。这个bug会出现在当dim不被nsq整除的情况。

在@nk2014yjdim=33数据集的例子中,nsq=8时的lsq=5,此时PQ的子空间维度为[5, 5, 5, 5, 5, 5, 3, 0],在处理0维度子空间的时候出现多余的内存复制导致overflow;但在nsq=6, lsq=6的情况下,PQ子空间维度为[6, 6, 6, 6, 6, 3]不存在0维度子空间所以可以正常执行。

麻烦请重新测试代码以验证问题是否解决,谢谢。

nk2014yj commented 1 day ago

puck支持3种距离计算,分别配置如下: L2 : whether_norm: false COS : 默认 IP : ip2cos (部分数据集支持) 你的配置,看着是期望ip距离。ip2cos,对数据集做了一个转换,用到的是 https://dl.acm.org/doi/pdf/10.5555/3327345.3327382 里的公式(4)。 这个转换在训练之前要做,转换后的数据默认存储在puck_index/all_data.feat.bin,这个时候文件的长度得 = 样本总数 (sizeof(float) (32 + 1)维度 + sizeof(int)) 。 转换的处理可以参考 https://github.com/baidu/puck/blob/0f03b409b259b3791a8c119bfbd769ef73e3e93f/ann-benchmarks/benchmark/algorithms/puck_inmem.py#L86

这个数据集的向量模不满足需求(<1.0),所以没办法直接计算ip距离。只能用cos距离作为近似距离计算,我测试cos是ok的,可以试试修改下参数(设置whether_norm: true,去掉ip2cos: 1) 。 我试着把数据集先按比例缩小,再用ip2cos。filter_nsq = 8 会出core,filter_nsq=6是ok的。应该是一些边界条件没做检查,需要定位一下问题&修复一下bug。

已修,推在分支fix_mem_corrupt。这个bug会出现在当dim不被nsq整除的情况。

在@nk2014yjdim=33数据集的例子中,nsq=8时的lsq=5,此时PQ的子空间维度为[5, 5, 5, 5, 5, 5, 3, 0],在处理0维度子空间的时候出现多余的内存复制导致overflow;但在nsq=6, lsq=6的情况下,PQ子空间维度为[6, 6, 6, 6, 6, 3]不存在0维度子空间所以可以正常执行。

麻烦请重新测试代码以验证问题是否解决,谢谢。

image

这边加个判断就好了,while(lsq * (nsq - 1) > dim){nsq --;}