secretflow / psi

The repo of Private Set Intersection(PSI) and Private Information Retrieval(PIR) from SecretFlow.
https://www.secretflow.org.cn/docs/psi
Apache License 2.0
26 stars 19 forks source link

[BUG]在3000万数据进行setup的时候报错 #116

Open yyf1986 opened 5 months ago

yyf1986 commented 5 months ago

Describe the bug 在对3000万数据进行setup的时候报错

2024-04-23 08:16:21.587 [info] [sender_kvdb.cc:InsertOrAssign:728] OPRF FullEvaluate and EncryptLabel Last batch count: 2, item_count:1000000
2024-04-23 08:16:21.590 [info] [sender_kvdb.cc:DispatchInsertOrAssign:517] Launching 1 insert-or-assign worker tasks
2024-04-23 08:16:35.774 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:0
2024-04-23 08:16:36.406 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:1
2024-04-23 08:16:36.880 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:2
2024-04-23 08:16:37.415 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:3
2024-04-23 08:16:38.189 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:4
2024-04-23 08:16:38.881 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:5
2024-04-23 08:16:39.464 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:6
2024-04-23 08:16:40.027 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:7
2024-04-23 08:16:40.573 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:8
2024-04-23 08:16:41.169 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:9
2024-04-23 08:16:41.890 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:10
2024-04-23 08:16:42.621 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:11
2024-04-23 08:16:43.269 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:12
2024-04-23 08:16:43.919 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:13
2024-04-23 08:16:44.596 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:14
2024-04-23 08:16:45.287 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:15
2024-04-23 08:16:45.963 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:16
2024-04-23 08:16:46.668 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:17
2024-04-23 08:16:47.281 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:18
2024-04-23 08:16:47.886 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:19
2024-04-23 08:16:48.537 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:20
2024-04-23 08:16:49.147 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:21
2024-04-23 08:16:49.713 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:22
2024-04-23 08:16:50.116 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:23
2024-04-23 08:16:50.744 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:24
2024-04-23 08:16:51.390 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:25
2024-04-23 08:16:51.984 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:26
2024-04-23 08:16:52.550 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:27
2024-04-23 08:16:53.149 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:28
2024-04-23 08:16:53.719 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:29
2024-04-23 08:16:54.331 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:30
2024-04-23 08:16:55.011 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:31
2024-04-23 08:16:55.600 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:32
2024-04-23 08:16:56.204 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:33
2024-04-23 08:16:56.807 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:34
2024-04-23 08:16:57.329 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:35
2024-04-23 08:16:57.859 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:36
2024-04-23 08:16:58.434 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:37
2024-04-23 08:16:59.026 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:38
2024-04-23 08:16:59.617 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:39
2024-04-23 08:17:00.172 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:40
2024-04-23 08:17:00.730 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:41
2024-04-23 08:17:01.334 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:42
2024-04-23 08:17:01.887 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:43
2024-04-23 08:17:02.535 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:44
2024-04-23 08:17:03.171 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:45
2024-04-23 08:17:03.812 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:46
2024-04-23 08:17:04.427 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:47
2024-04-23 08:17:05.092 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:48
2024-04-23 08:17:05.743 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:49
2024-04-23 08:17:06.360 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:50
2024-04-23 08:17:06.926 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:51
2024-04-23 08:17:07.504 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:52
2024-04-23 08:17:08.078 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:53
2024-04-23 08:17:08.649 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:54
2024-04-23 08:17:09.041 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:55
2024-04-23 08:17:09.584 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:56
2024-04-23 08:17:10.106 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:57
2024-04-23 08:17:10.586 [info] [sender_kvdb.cc:InsertOrAssignWorker:433] *** step leveldb put duration:34812.315606
2024-04-23 08:17:10.671 [info] [sender_kvdb.cc:DispatchInsertOrAssign:535] Finished insert-or-assign worker tasks
2024-04-23 08:17:10.676 [info] [sender_kvdb.cc:InsertOrAssign:841] Finished inserting 1000000 items in SenderDB
2024-04-23 08:17:10.972 [info] [pir.cc:LabeledPirSetup:268] finish bucket:28
2024-04-23 08:17:11.088 [info] [pir.cc:LabeledPirSetup:240] bucket:29 bucket_setup_path:/data/3000w_setup_path/bucket_29
2024-04-23 08:17:11.348 [info] [ecdh_oprf_selector.cc:CreateEcdhOprfServer:33] use fourq
2024-04-23 08:17:11.419 [info] [leveldb_kvstore.cc:Get:73] key not found
2024-04-23 08:17:11.419 [info] [sender_kvdb.cc:SenderKvDB:597] key item_count no value
2024-04-23 08:17:11.590 [info] [sender_kvdb.cc:InsertOrAssign:734] OPRF FullEvaluate and EncryptLabel batch_count: 0
2024-04-23 08:17:38.348 [info] [sender_kvdb.cc:InsertOrAssign:734] OPRF FullEvaluate and EncryptLabel batch_count: 1
2024-04-23 08:17:49.386 [info] [sender_kvdb.cc:InsertOrAssign:728] OPRF FullEvaluate and EncryptLabel Last batch count: 2, item_count:1000000
2024-04-23 08:17:49.389 [info] [sender_kvdb.cc:DispatchInsertOrAssign:517] Launching 1 insert-or-assign worker tasks
2024-04-23 08:18:01.863 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:0
2024-04-23 08:18:02.475 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:1
2024-04-23 08:18:02.978 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:2
2024-04-23 08:18:03.521 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:3
2024-04-23 08:18:04.268 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:4
2024-04-23 08:18:04.927 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:5
2024-04-23 08:18:05.575 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:6
2024-04-23 08:18:06.305 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:7
2024-04-23 08:18:06.978 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:8
2024-04-23 08:18:07.626 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:9
2024-04-23 08:18:08.255 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:10
2024-04-23 08:18:08.830 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:11
2024-04-23 08:18:09.456 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:12
2024-04-23 08:18:10.100 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:13
2024-04-23 08:18:10.683 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:14
2024-04-23 08:18:11.274 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:15
2024-04-23 08:18:11.890 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:16
2024-04-23 08:18:12.543 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:17
2024-04-23 08:18:13.185 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:18
2024-04-23 08:18:13.778 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:19
2024-04-23 08:18:14.323 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:20
2024-04-23 08:18:14.955 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:21
2024-04-23 08:18:15.522 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:22
2024-04-23 08:18:16.102 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:23
2024-04-23 08:18:16.698 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:24
2024-04-23 08:18:17.277 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:25
2024-04-23 08:18:17.829 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:26
2024-04-23 08:18:18.394 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:27
2024-04-23 08:18:18.923 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:28
2024-04-23 08:18:19.452 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:29
2024-04-23 08:18:20.009 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:30
2024-04-23 08:18:20.619 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:31
2024-04-23 08:18:21.157 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:32
2024-04-23 08:18:21.694 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:33
2024-04-23 08:18:22.319 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:34
2024-04-23 08:18:22.933 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:35
2024-04-23 08:18:23.501 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:36
2024-04-23 08:18:24.184 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:37
2024-04-23 08:18:24.841 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:38
2024-04-23 08:18:25.497 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:39
2024-04-23 08:18:26.114 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:40
2024-04-23 08:18:26.744 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:41
2024-04-23 08:18:27.325 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:42
2024-04-23 08:18:27.913 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:43
2024-04-23 08:18:28.492 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:44
2024-04-23 08:18:29.012 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:45
2024-04-23 08:18:29.578 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:46
2024-04-23 08:18:30.162 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:47
2024-04-23 08:18:30.716 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:48
2024-04-23 08:18:31.270 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:49
2024-04-23 08:18:31.854 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:50
2024-04-23 08:18:32.440 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:51
2024-04-23 08:18:33.046 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:52
2024-04-23 08:18:33.626 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:53
2024-04-23 08:18:34.191 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:54
2024-04-23 08:18:34.816 [info] [sender_kvdb.cc:InsertOrAssignWorker:417] Polynomial Interpolate and HE Plaintext Encode, bundle_indx:0, store_idx:55
Traceback (most recent call last):
  File "pir_setup.py", line 61, in <module>
  File "spu/pir.py", line 31, in pir_setup
RuntimeError: tried to interpolate at repeated points
[24825] Failed to execute script 'pir_setup' due to unhandled exception!

在进行3000万数据setup的时候,在第bucket_29的时候出错了。 而且指定bucket_size没有效果,都是固定的100万。 1000万,2000万数据都可以正常进行setup操作。 数据已经进行去重操作。

python /pir_setup.py \
--setup-data /opt/3000w.csv  \
--key-columns 'id' \
--label-column 'fenshu' \
--pir-setup-path /data/3000w_setup_path \
--pir-oprf-key-path /oprf_key.bin 

执行命令

The version of some packages SecretFlow: 1.3.0.dev20231109
SPU: 0.6.0b0
Ray: 2.2.0
Jax: 0.4.12
Jaxlib: 0.4.12
TensorFlow: 2.11.1
PyTorch: 2.0.0+cpu

6fj commented 5 months ago

请问可以用一下最新的代码吗,目前最新的secretflow版本是:1.5.0b0

yyf1986 commented 5 months ago

@6fj 我试试

yyf1986 commented 5 months ago

@6fj 新版本的可以了。另外有没有pir的一些性能数据阿

yyf1986 commented 5 months ago

setup预处理的时候,如何降低处理时间呢?

Chrisdehe commented 5 months ago

我们正在关注此实现,敬请期待

6fj commented 4 months ago

可以尝试将label_max_len设置小一点