jmazanec15
commented
3 years ago Hi @Hronom,
Taking a look. Will update this thread when I find something.
Open Hronom opened 3 years ago
jmazanec15
commented
3 years ago Hi @Hronom,
Taking a look. Will update this thread when I find something.
jmazanec15
commented
3 years ago So, for search, ES will send the request to 6 shards (could be either primary or replica) depending on routing. Then, like you said in the expected behavior, each segment will return k results and then each shard will return top size results and ES will return the top size results from those.
One potential difference is that the primaries/replicas have different number of segments. Could you run GET /_cat/segments and see if there is any difference in the segments between each primary and the corresponding replica?
Another potential difference is that if you are indexing with replicas enabled, the replicas will build their own segments' graphs and these graphs will be different from the primaries. To make it so that the replicas have the same graphs as the primaries, you could disable replicas when indexing, and then enable them after indexing workload finishes. This will copy the primaries to the replicas so that they do not have different graphs. However, to be able to do this depends on your workflow. If you are continuously updating the index, this probably won't work.
A few other questions:
routing isn't enabled? Do you have a recall number that shows how significantly the results vary?
Hronom
commented
3 years ago 108_232 docs, if we disable routing start get random hits, example variation: 1543 vs 1854:
"hits" : {
"total" : {
"value" : 1854,
"relation" : "eq"
},108_232 docs. We also have for now only one vector field12 data nodes with index.routing.allocation.total_shards_per_node set to 1. Also we have 3 mastersAlso we use post filter in our query. So this is why we set k to 30 to make sure that we get enough results for size 3
Hronom
commented
3 years ago Also how difference in segments affects results? I'm assume that there results merged from each segment so you always get constant list, since during merging you sort by score. Or this happens because each graph in each segment has local score? So differently formed segments forms differently formed graphs with different local scores, is this the case?
During indexation we do routing and use custom id, so all documents places on specific shards.
jmazanec15
commented
3 years ago The total hits returned is the number of results returned for all of the segments. Assume that a replica shard has 4 segments (each has a graph) with 25,000 documents each, and a primary has 2 segments with 50,000 documents each. If k=30, searching the replica will return 120 results, while searching the primary would return only 60 results (30 from each). If size=3, the top 3 results would be returned from the primary or replica.
I am curious what the number of segments are for your primary and replica shards. Could you provide that information?
Also, how are you measuring the quality of the search (i.e. recall)? Is there a noticeable performance drop in quality of the search when using one routing compared to another?
Hronom
commented
3 years ago I am curious what the number of segments are for your primary and replica shards. Could you provide that information? Here what I get from
GET /<index name>/_segments(I removed segments info and nodes names):{ "_shards" : { "total" : 12, "successful" : 12, "failed" : 0 }, "indices" : { "projects" : { "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 9, "num_search_segments" : 15 }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 11, "num_search_segments" : 6 } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 18, "num_search_segments" : 17 }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 19, "num_search_segments" : 14 } ], "2" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 13, "num_search_segments" : 8 }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 16, "num_search_segments" : 16 } ], "3" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 16, "num_search_segments" : 17 }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 12, "num_search_segments" : 18 } ], "4" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 14, "num_search_segments" : 16 }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 19, "num_search_segments" : 15 } ], "5" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "-" }, "num_committed_segments" : 12, "num_search_segments" : 15, }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "-" }, "num_committed_segments" : 19, "num_search_segments" : 22 } ] } } } }Also, how are you measuring the quality of the search (i.e. recall)? Is there a noticeable performance drop in quality of the search when using one routing compared to another?
This is not done yet, we done only offline evaluation(out from Elasticsearch) to confirm that vector space provides good results.
Also our index not static, we receive updates continuously
jmazanec15
commented
3 years ago Hi @Hronom
Could you provide output of _cat/segments? That will allow me to compare primaries to replicas.
I am curious if the difference in hit size will impact the recall significantly, with the post filter enabled. If it does, you could always increase k until you consistently get the number you are looking for.
Jack
Hronom
commented
3 years ago Hello @jmazanec15
index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound
unknown 0 r x.x.x.x _6b9 8181 655 337 10.6mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6ba 8182 18031 5486 194.3mb 16598 true true 8.6.2 false
unknown 0 r x.x.x.x _6bl 8193 243 63 3.3mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bm 8194 278 49 3.6mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bn 8195 287 31 3.6mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bo 8196 284 24 3.6mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bp 8197 274 5 3.3mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bq 8198 312 12 3.7mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6br 8199 313 6 3.7mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bs 8200 314 83 4.4mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6bx 8205 1 6 7.5kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c4 8212 578 66 7.2mb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c5 8213 10 3 167.3kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c6 8214 6 1 99.7kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c7 8215 9 75 528.6kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c8 8216 3 84 606.5kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6c9 8217 26 53 702.3kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6ca 8218 1 0 5.2kb 0 true false 8.6.2 true
unknown 0 r x.x.x.x _6dk 8264 2084 2390 43.9mb 24022 false true 8.6.2 false
unknown 0 r x.x.x.x _6f7 8323 2037 7399 80.9mb 22990 false true 8.6.2 false
unknown 0 r x.x.x.x _6fr 8343 50 865 6mb 70836 false true 8.6.2 true
unknown 0 r x.x.x.x _6g2 8354 296 788 10mb 19172 false true 8.6.2 true
unknown 0 r x.x.x.x _6gc 8364 17 802 7mb 40060 false true 8.6.2 true
unknown 0 r x.x.x.x _6gl 8373 301 1503 12.2mb 19364 false true 8.6.2 true
unknown 0 r x.x.x.x _6gv 8383 261 817 8mb 18996 false true 8.6.2 true
unknown 0 r x.x.x.x _6h6 8394 226 935 9.6mb 191044 false true 8.6.2 true
unknown 0 r x.x.x.x _6hh 8405 46 1018 7.8mb 63724 false true 8.6.2 true
unknown 0 r x.x.x.x _6hp 8413 462 1043 12.9mb 21732 false true 8.6.2 true
unknown 0 r x.x.x.x _6hz 8423 251 1540 15.6mb 18660 false true 8.6.2 true
unknown 0 r x.x.x.x _6i9 8433 113 878 7.3mb 97764 false true 8.6.2 true
unknown 0 r x.x.x.x _6ik 8444 254 1056 9.6mb 185076 false true 8.6.2 true
unknown 0 r x.x.x.x _6il 8445 33 103 1mb 42820 false true 8.6.2 true
unknown 0 r x.x.x.x _6im 8446 45 156 1.9mb 56188 false true 8.6.2 true
unknown 0 r x.x.x.x _6in 8447 10 94 954.7kb 26452 false true 8.6.2 true
unknown 0 r x.x.x.x _6io 8448 0 8 137.2kb 19980 false true 8.6.2 true
unknown 0 r x.x.x.x _6ip 8449 39 183 2mb 49164 false true 8.6.2 true
unknown 0 r x.x.x.x _6iq 8450 11 51 642.8kb 23188 false true 8.6.2 true
unknown 0 r x.x.x.x _6ir 8451 43 107 1.2mb 48620 false true 8.6.2 true
unknown 0 p x.x.x.x _7ai 9450 23469 982 203.2mb 16590 true true 8.6.2 false
unknown 0 p x.x.x.x _7cb 9515 32 48 754.4kb 42628 true true 8.6.2 true
unknown 0 p x.x.x.x _7d0 9540 39 169 1.4mb 50468 true true 8.6.2 true
unknown 0 p x.x.x.x _7d3 9543 24 232 1.9mb 42628 true true 8.6.2 true
unknown 0 p x.x.x.x _7d4 9544 0 35 375kb 0 true false 8.6.2 true
unknown 0 p x.x.x.x _7d5 9545 25 197 1.6mb 39452 true true 8.6.2 true
unknown 0 p x.x.x.x _7d6 9546 0 12 178.9kb 0 true false 8.6.2 true
unknown 0 p x.x.x.x _7d7 9547 58 150 1.7mb 61356 true true 8.6.2 true
unknown 0 p x.x.x.x _7d8 9548 0 17 216.1kb 0 true false 8.6.2 true
unknown 0 p x.x.x.x _7d9 9549 30 10 341.1kb 32716 true true 8.6.2 true
unknown 0 p x.x.x.x _7da 9550 68 177 1.8mb 71820 true true 8.6.2 true
unknown 0 p x.x.x.x _7dc 9552 675 442 9.4mb 20772 true true 8.6.2 true
unknown 0 p x.x.x.x _7dd 9553 40 145 1.3mb 46196 true true 8.6.2 true
unknown 0 p x.x.x.x _7de 9554 29 123 1.3mb 43276 true true 8.6.2 true
unknown 0 p x.x.x.x _7df 9555 23 135 1.4mb 37476 true true 8.6.2 true
unknown 0 p x.x.x.x _7dg 9556 25 51 822.5kb 37524 true true 8.6.2 true
unknown 0 p x.x.x.x _7dh 9557 2 18 244kb 15292 true true 8.6.2 true
unknown 0 p x.x.x.x _7di 9558 21 25 510.6kb 30508 false true 8.6.2 true
unknown 0 p x.x.x.x _7dj 9559 0 43 461.3kb 15508 false true 8.6.2 true
unknown 0 p x.x.x.x _7dk 9560 0 28 324.9kb 19052 false true 8.6.2 true
unknown 0 p x.x.x.x _7dl 9561 3 60 634.2kb 20932 false true 8.6.2 true
unknown 0 p x.x.x.x _7dm 9562 47 116 1.3mb 51764 false true 8.6.2 true
unknown 1 r x.x.x.x _64j 7939 16195 5884 179.6mb 18318 true true 8.6.2 false
unknown 1 r x.x.x.x _672 8030 1080 1506 27.9mb 0 true false 8.6.2 false
unknown 1 r x.x.x.x _673 8031 11 143 1023.4kb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _678 8036 7 157 1014.8kb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67a 8038 16 158 1.1mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67m 8050 3195 3202 63.1mb 19518 true true 8.6.2 false
unknown 1 r x.x.x.x _67n 8051 256 144 4mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67p 8053 245 105 3.5mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67t 8057 47 48 1mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67u 8058 95 62 1.4mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _67w 8060 1530 1272 26.7mb 0 true false 8.6.2 false
unknown 1 r x.x.x.x _67y 8062 9 168 1.5mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _686 8070 125 409 3.3mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _687 8071 5 162 1.1mb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _688 8072 9 38 425.3kb 0 true false 8.6.2 true
unknown 1 r x.x.x.x _691 8101 468 1427 17.2mb 20566 false true 8.6.2 true
unknown 1 r x.x.x.x _69k 8120 3110 1799 47.9mb 22998 false true 8.6.2 false
unknown 1 r x.x.x.x _69u 8130 60 695 4.7mb 73604 false true 8.6.2 true
unknown 1 r x.x.x.x _6a4 8140 42 980 6.7mb 67572 false true 8.6.2 true
unknown 1 r x.x.x.x _6af 8151 71 839 6.5mb 66620 false true 8.6.2 true
unknown 1 r x.x.x.x _6ao 8160 124 854 6.4mb 109724 false true 8.6.2 true
unknown 1 r x.x.x.x _6ay 8170 147 651 6mb 123108 false true 8.6.2 true
unknown 1 r x.x.x.x _6b8 8180 53 1016 7.4mb 58308 false true 8.6.2 true
unknown 1 r x.x.x.x _6bk 8192 97 983 7.9mb 88980 false true 8.6.2 true
unknown 1 r x.x.x.x _6bs 8200 108 1458 11.8mb 105236 false true 8.6.2 true
unknown 1 r x.x.x.x _6c2 8210 218 1148 10.8mb 172492 false true 8.6.2 true
unknown 1 r x.x.x.x _6c3 8211 39 273 2.2mb 50484 false true 8.6.2 true
unknown 1 r x.x.x.x _6c4 8212 59 215 2.4mb 64252 false true 8.6.2 true
unknown 1 r x.x.x.x _6c5 8213 1 7 141.7kb 19956 false true 8.6.2 true
unknown 1 r x.x.x.x _6c6 8214 46 95 1.3mb 52636 false true 8.6.2 true
unknown 1 r x.x.x.x _6c7 8215 24 10 413.9kb 35780 false true 8.6.2 true
unknown 1 r x.x.x.x _6c8 8216 18 109 1mb 29900 false true 8.6.2 true
unknown 1 r x.x.x.x _6c9 8217 42 26 593.9kb 46028 false true 8.6.2 true
unknown 1 r x.x.x.x _6ca 8218 46 108 1mb 50388 false true 8.6.2 true
unknown 1 r x.x.x.x _6cb 8219 45 2 607.5kb 49764 false true 8.6.2 true
unknown 1 p x.x.x.x _741 9217 973 384 15.5mb 22102 true true 8.6.2 true
unknown 1 p x.x.x.x _742 9218 20783 1712 182.5mb 18198 true true 8.6.2 false
unknown 1 p x.x.x.x _76q 9314 738 731 13.9mb 21116 true true 8.6.2 true
unknown 1 p x.x.x.x _77v 9355 911 1154 17.4mb 20732 true true 8.6.2 true
unknown 1 p x.x.x.x _78n 9383 20 86 1mb 38436 true true 8.6.2 true
unknown 1 p x.x.x.x _78p 9385 32 146 1.3mb 49708 true true 8.6.2 true
unknown 1 p x.x.x.x _78r 9387 11 130 1.2mb 27652 true true 8.6.2 true
unknown 1 p x.x.x.x _78s 9388 90 34 1.2mb 85364 true true 8.6.2 true
unknown 1 p x.x.x.x _78v 9391 42 283 2.2mb 51652 true true 8.6.2 true
unknown 1 p x.x.x.x _78w 9392 52 303 2.7mb 60356 true true 8.6.2 true
unknown 1 p x.x.x.x _78y 9394 48 121 1.4mb 55268 true true 8.6.2 true
unknown 1 p x.x.x.x _792 9398 23 110 1.1mb 33876 true true 8.6.2 true
unknown 1 p x.x.x.x _793 9399 351 377 6.7mb 18348 true true 8.6.2 true
unknown 1 p x.x.x.x _794 9400 53 39 738.7kb 54564 true true 8.6.2 true
unknown 1 p x.x.x.x _795 9401 28 91 863.1kb 36252 true true 8.6.2 true
unknown 1 p x.x.x.x _796 9402 8 0 140.6kb 20876 false true 8.6.2 true
unknown 1 p x.x.x.x _797 9403 46 2 619.5kb 50500 false true 8.6.2 true
unknown 2 p x.x.x.x _7h7 9691 21998 1937 197.5mb 16526 true true 8.6.2 false
unknown 2 p x.x.x.x _7jj 9775 448 195 5.9mb 22164 true true 8.6.2 true
unknown 2 p x.x.x.x _7k4 9796 570 282 8.5mb 21924 true true 8.6.2 true
unknown 2 p x.x.x.x _7lp 9853 808 1168 17.2mb 21004 true true 8.6.2 true
unknown 2 p x.x.x.x _7me 9878 100 466 3.7mb 89420 true true 8.6.2 true
unknown 2 p x.x.x.x _7mh 9881 25 109 986kb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mi 9882 37 11 496.3kb 42348 true true 8.6.2 true
unknown 2 p x.x.x.x _7mj 9883 16 214 2.1mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mk 9884 20 255 2.1mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mm 9886 33 178 1.5mb 44604 true true 8.6.2 true
unknown 2 p x.x.x.x _7mn 9887 24 398 3.5mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mo 9888 0 274 2.3mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mp 9889 16 231 2.1mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mq 9890 122 53 1.6mb 96212 true true 8.6.2 true
unknown 2 p x.x.x.x _7mr 9891 41 61 880.6kb 45756 true true 8.6.2 true
unknown 2 p x.x.x.x _7ms 9892 26 200 1.5mb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mt 9893 27 79 882.5kb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mu 9894 47 273 2.4mb 56332 true true 8.6.2 true
unknown 2 p x.x.x.x _7mv 9895 6 68 621.5kb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mw 9896 30 209 1.6mb 45476 true true 8.6.2 true
unknown 2 p x.x.x.x _7mx 9897 52 28 813.2kb 55244 true true 8.6.2 true
unknown 2 p x.x.x.x _7my 9898 21 75 798.7kb 0 true false 8.6.2 true
unknown 2 p x.x.x.x _7mz 9899 178 3 1.8mb 133228 false true 8.6.2 true
unknown 2 p x.x.x.x _7n0 9900 25 129 1.1mb 34308 false true 8.6.2 true
unknown 2 p x.x.x.x _7n1 9901 0 31 333kb 15164 false true 8.6.2 true
unknown 2 p x.x.x.x _7n2 9902 46 24 800.2kb 51724 false true 8.6.2 true
unknown 2 r x.x.x.x _6mi 8586 4007 8602 114.1mb 0 true false 8.6.2 false
unknown 2 r x.x.x.x _6og 8656 15095 13014 215.9mb 0 true false 8.6.2 false
unknown 2 r x.x.x.x _6p1 8677 893 2558 36.1mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6pu 8706 67 913 8mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6q4 8716 28 1114 9.3mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6qf 8727 30 1166 9.6mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6qp 8737 49 1235 8.2mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6qz 8747 1805 2236 38.8mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6r2 8750 223 524 6.4mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6r9 8757 870 1179 20.1mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rb 8759 0 28 254.4kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rc 8760 0 34 376.8kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rd 8761 0 9 135.9kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6ri 8766 22 71 932.8kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rk 8768 1277 2656 36.4mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rl 8769 17 181 1.5mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rm 8770 15 69 629.5kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rn 8771 13 242 2mb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6ro 8772 10 94 991.3kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rp 8773 1 5 104.1kb 0 true false 8.6.2 true
unknown 2 r x.x.x.x _6rs 8776 22305 1932 201mb 16542 false true 8.6.2 false
unknown 2 r x.x.x.x _6t6 8826 205 1012 8.6mb 183916 false true 8.6.2 true
unknown 2 r x.x.x.x _6tg 8836 261 1197 12.4mb 19260 false true 8.6.2 true
unknown 2 r x.x.x.x _6tr 8847 49 1015 7.5mb 56268 false true 8.6.2 true
unknown 2 r x.x.x.x _6ub 8867 649 2191 24.7mb 20372 false true 8.6.2 true
unknown 2 r x.x.x.x _6ve 8906 46 1649 12mb 49732 false true 8.6.2 true
unknown 2 r x.x.x.x _6vo 8916 423 4929 39.9mb 20380 false true 8.6.2 false
unknown 2 r x.x.x.x _6vy 8926 67 380 2.7mb 73540 false true 8.6.2 true
unknown 2 r x.x.x.x _6w4 8932 32 422 3.4mb 49020 false true 8.6.2 true
unknown 2 r x.x.x.x _6w6 8934 2 336 2.9mb 26940 false true 8.6.2 true
unknown 2 r x.x.x.x _6wa 8938 70 846 7.1mb 70668 false true 8.6.2 true
unknown 2 r x.x.x.x _6we 8942 42 266 2.3mb 49476 false true 8.6.2 true
unknown 2 r x.x.x.x _6wi 8946 267 1343 13.1mb 17508 false true 8.6.2 true
unknown 2 r x.x.x.x _6wj 8947 38 237 2mb 45652 false true 8.6.2 true
unknown 2 r x.x.x.x _6wk 8948 45 55 1mb 56620 false true 8.6.2 true
unknown 2 r x.x.x.x _6wl 8949 0 25 322.4kb 23012 false true 8.6.2 true
unknown 2 r x.x.x.x _6wm 8950 34 88 1.1mb 33292 false true 8.6.2 true
unknown 3 r x.x.x.x _63q 7910 1058 738 18.9mb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _63r 7911 16660 5709 181.9mb 18366 true true 8.6.2 false
unknown 3 r x.x.x.x _64c 7932 497 1090 14.8mb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _64j 7939 272 105 4mb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _64k 7940 281 237 5.1mb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _64m 7942 1111 1003 21.6mb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _64w 7952 1862 884 28.8mb 0 true false 8.6.2 false
unknown 3 r x.x.x.x _652 7958 16 37 523.5kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _653 7959 2 25 199.1kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _654 7960 15 63 527.5kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _656 7962 7 48 355.6kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _657 7963 66 9 727.4kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _658 7964 17 52 609kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _659 7965 3 37 298kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _65a 7966 6 19 231.6kb 0 true false 8.6.2 true
unknown 3 r x.x.x.x _66n 8015 3410 3564 70.2mb 15958 false true 8.6.2 false
unknown 3 r x.x.x.x _66w 8024 748 1640 22.4mb 21790 false true 8.6.2 true
unknown 3 r x.x.x.x _68a 8074 793 1265 20.6mb 22582 false true 8.6.2 true
unknown 3 r x.x.x.x _6a8 8144 1615 10012 85.6mb 23940 false true 8.6.2 false
unknown 3 r x.x.x.x _6ae 8150 17 227 1.7mb 32020 false true 8.6.2 true
unknown 3 r x.x.x.x _6af 8151 12 227 1.5mb 28412 false true 8.6.2 true
unknown 3 r x.x.x.x _6ai 8154 62 451 2.6mb 62340 false true 8.6.2 true
unknown 3 r x.x.x.x _6ak 8156 12 166 1mb 27508 false true 8.6.2 true
unknown 3 r x.x.x.x _6an 8159 7 166 1.3mb 25092 false true 8.6.2 true
unknown 3 r x.x.x.x _6at 8165 104 591 4.2mb 86020 false true 8.6.2 true
unknown 3 r x.x.x.x _6av 8167 31 122 1.2mb 41244 false true 8.6.2 true
unknown 3 r x.x.x.x _6b2 8174 249 722 7.4mb 186628 false true 8.6.2 true
unknown 3 r x.x.x.x _6b3 8175 24 17 480.1kb 35268 false true 8.6.2 true
unknown 3 r x.x.x.x _6b4 8176 11 45 608.8kb 27724 false true 8.6.2 true
unknown 3 r x.x.x.x _6b5 8177 37 92 1.1mb 45820 false true 8.6.2 true
unknown 3 r x.x.x.x _6b6 8178 39 7 469.6kb 40924 false true 8.6.2 true
unknown 3 r x.x.x.x _6b7 8179 44 83 1.2mb 50524 false true 8.6.2 true
unknown 3 r x.x.x.x _6b8 8180 49 141 1.2mb 48444 false true 8.6.2 true
unknown 3 p x.x.x.x _72n 9167 19585 4074 196.5mb 16470 true true 8.6.2 false
unknown 3 p x.x.x.x _73h 9197 1949 650 29.3mb 21222 true true 8.6.2 false
unknown 3 p x.x.x.x _75t 9281 1226 1034 21.3mb 22142 true true 8.6.2 true
unknown 3 p x.x.x.x _77c 9336 466 335 6.8mb 20252 true true 8.6.2 true
unknown 3 p x.x.x.x _782 9362 17 106 966.9kb 31356 true true 8.6.2 true
unknown 3 p x.x.x.x _784 9364 19 73 821.4kb 32916 true true 8.6.2 true
unknown 3 p x.x.x.x _786 9366 88 44 982.3kb 65980 true true 8.6.2 true
unknown 3 p x.x.x.x _787 9367 11 81 785.9kb 28092 true true 8.6.2 true
unknown 3 p x.x.x.x _789 9369 51 61 1mb 55388 true true 8.6.2 true
unknown 3 p x.x.x.x _78a 9370 43 39 835kb 50548 true true 8.6.2 true
unknown 3 p x.x.x.x _78b 9371 25 47 767.5kb 40164 true true 8.6.2 true
unknown 3 p x.x.x.x _78f 9375 35 91 1.2mb 44868 true true 8.6.2 true
unknown 3 p x.x.x.x _78h 9377 239 289 4.5mb 174940 true true 8.6.2 true
unknown 3 p x.x.x.x _78i 9378 39 43 749kb 45548 true true 8.6.2 true
unknown 3 p x.x.x.x _78j 9379 47 12 614.6kb 50396 true true 8.6.2 true
unknown 3 p x.x.x.x _78k 9380 18 34 547.6kb 31276 true true 8.6.2 true
unknown 3 p x.x.x.x _78l 9381 0 6 126.6kb 0 true false 8.6.2 true
unknown 3 p x.x.x.x _78m 9382 27 28 597.8kb 36756 false true 8.6.2 true
unknown 3 p x.x.x.x _78n 9383 0 12 178.7kb 15484 false true 8.6.2 true
unknown 3 p x.x.x.x _78o 9384 39 191 1.4mb 40684 false true 8.6.2 true
unknown 4 r x.x.x.x _69k 8120 2530 1721 47.3mb 22982 true true 8.6.2 false
unknown 4 r x.x.x.x _69l 8121 16485 3587 156.6mb 16262 true true 8.6.2 false
unknown 4 r x.x.x.x _69m 8122 11 109 956.3kb 0 true false 8.6.2 true
unknown 4 r x.x.x.x _69n 8123 7 17 202.2kb 0 true false 8.6.2 true
unknown 4 r x.x.x.x _69o 8124 1 0 60.2kb 0 true false 8.6.2 true
unknown 4 r x.x.x.x _6b1 8173 1801 1881 36.5mb 23838 false true 8.6.2 false
unknown 4 r x.x.x.x _6bu 8202 1031 3300 38.4mb 21702 false true 8.6.2 false
unknown 4 r x.x.x.x _6d8 8252 449 920 12.4mb 20932 false true 8.6.2 true
unknown 4 r x.x.x.x _6ec 8292 1157 8388 72.2mb 22300 false true 8.6.2 false
unknown 4 r x.x.x.x _6em 8302 44 238 1.6mb 48404 false true 8.6.2 true
unknown 4 r x.x.x.x _6et 8309 17 208 1.6mb 32316 false true 8.6.2 true
unknown 4 r x.x.x.x _6ew 8312 80 532 3.6mb 72660 false true 8.6.2 true
unknown 4 r x.x.x.x _6f6 8322 141 584 5mb 113372 false true 8.6.2 true
unknown 4 r x.x.x.x _6f8 8324 26 205 1.9mb 40292 false true 8.6.2 true
unknown 4 r x.x.x.x _6f9 8325 3 195 1.7mb 25292 false true 8.6.2 true
unknown 4 r x.x.x.x _6fg 8332 27 153 1.5mb 36276 false true 8.6.2 true
unknown 4 r x.x.x.x _6fh 8333 160 785 6.8mb 139684 false true 8.6.2 true
unknown 4 r x.x.x.x _6fi 8334 33 138 1.2mb 45492 false true 8.6.2 true
unknown 4 r x.x.x.x _6fj 8335 29 155 1.2mb 42308 false true 8.6.2 true
unknown 4 r x.x.x.x _6fk 8336 45 128 1.2mb 51892 false true 8.6.2 true
unknown 4 r x.x.x.x _6fl 8337 39 95 1mb 47492 false true 8.6.2 true
unknown 4 r x.x.x.x _6fm 8338 47 168 1.5mb 53924 false true 8.6.2 true
unknown 4 r x.x.x.x _6fn 8339 29 121 1.1mb 39180 false true 8.6.2 true
unknown 4 r x.x.x.x _6fo 8340 76 37 1.1mb 71164 false true 8.6.2 true
unknown 4 p x.x.x.x _7av 9463 21689 2186 197.9mb 16502 true true 8.6.2 false
unknown 4 p x.x.x.x _7dj 9559 672 901 15.9mb 22332 true true 8.6.2 true
unknown 4 p x.x.x.x _7e5 9581 1075 780 17.4mb 21716 true true 8.6.2 true
unknown 4 p x.x.x.x _7eu 9606 55 13 661.2kb 52884 true true 8.6.2 true
unknown 4 p x.x.x.x _7f8 9620 6 41 365.5kb 21724 true true 8.6.2 true
unknown 4 p x.x.x.x _7fa 9622 9 248 2.2mb 25732 true true 8.6.2 true
unknown 4 p x.x.x.x _7fb 9623 9 49 608.1kb 28244 true true 8.6.2 true
unknown 4 p x.x.x.x _7fd 9625 26 40 619.3kb 37044 true true 8.6.2 true
unknown 4 p x.x.x.x _7fg 9628 0 17 212.7kb 0 true false 8.6.2 true
unknown 4 p x.x.x.x _7fi 9630 14 56 623.9kb 30284 true true 8.6.2 true
unknown 4 p x.x.x.x _7fk 9632 2 30 369.6kb 19172 true true 8.6.2 true
unknown 4 p x.x.x.x _7fo 9636 501 848 10.6mb 19188 true true 8.6.2 true
unknown 4 p x.x.x.x _7fp 9637 35 95 1mb 44444 true true 8.6.2 true
unknown 4 p x.x.x.x _7fq 9638 51 168 1.6mb 56964 true true 8.6.2 true
unknown 4 p x.x.x.x _7fr 9639 4 83 635.2kb 23012 true true 8.6.2 true
unknown 4 p x.x.x.x _7fs 9640 12 36 443.8kb 27684 true true 8.6.2 true
unknown 4 p x.x.x.x _7ft 9641 12 2 179.2kb 23108 false true 8.6.2 true
unknown 4 p x.x.x.x _7fu 9642 77 37 1.1mb 71924 false true 8.6.2 true
unknown 5 p x.x.x.x _7fb 9623 19704 5572 209.9mb 16566 true true 8.6.2 false
unknown 5 p x.x.x.x _7hs 9712 667 622 13.3mb 20846 true true 8.6.2 true
unknown 5 p x.x.x.x _7i1 9721 633 472 12.2mb 24358 true true 8.6.2 true
unknown 5 p x.x.x.x _7j7 9763 1729 1314 32.7mb 23574 true true 8.6.2 false
unknown 5 p x.x.x.x _7l6 9834 860 584 14mb 20836 true true 8.6.2 true
unknown 5 p x.x.x.x _7mw 9896 94 5 1003.6kb 75972 true true 8.6.2 true
unknown 5 p x.x.x.x _7n3 9903 37 290 2.8mb 58436 true true 8.6.2 true
unknown 5 p x.x.x.x _7n6 9906 241 509 5.8mb 184548 true true 8.6.2 true
unknown 5 p x.x.x.x _7n7 9907 32 36 397kb 36348 true true 8.6.2 true
unknown 5 p x.x.x.x _7n8 9908 32 23 457.8kb 38020 true true 8.6.2 true
unknown 5 p x.x.x.x _7na 9910 34 251 2.3mb 44012 true true 8.6.2 true
unknown 5 p x.x.x.x _7nc 9912 12 299 2.5mb 30348 true true 8.6.2 true
unknown 5 p x.x.x.x _7nd 9913 0 3 104.5kb 0 true false 8.6.2 true
unknown 5 p x.x.x.x _7ng 9916 813 747 14.1mb 21548 true true 8.6.2 true
unknown 5 p x.x.x.x _7nh 9917 16 26 372.1kb 29596 true true 8.6.2 true
unknown 5 p x.x.x.x _7ni 9918 0 1 7kb 2436 false true 8.6.2 true
unknown 5 p x.x.x.x _7nj 9919 15 66 626.8kb 27684 false true 8.6.2 true
unknown 5 p x.x.x.x _7nk 9920 49 48 849.7kb 49196 false true 8.6.2 true
unknown 5 r x.x.x.x _6gm 8374 7293 4190 125.2mb 17854 true true 8.6.2 false
unknown 5 r x.x.x.x _6go 8376 9551 3693 82.1mb 17790 true true 8.6.2 false
unknown 5 r x.x.x.x _6h8 8396 1117 951 22.3mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6hi 8406 542 806 13.5mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6hs 8416 658 787 14.5mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6i4 8428 1302 1330 27.5mb 0 true false 8.6.2 false
unknown 5 r x.x.x.x _6ic 8436 2028 1119 32.8mb 0 true false 8.6.2 false
unknown 5 r x.x.x.x _6id 8437 171 73 2.5mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6im 8446 14 269 2.2mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6iw 8456 60 389 3.3mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6j0 8460 25 230 2.3mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6j6 8466 109 894 8.9mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6jg 8476 110 1283 10.3mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6jr 8487 69 753 6.8mb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6js 8488 3 0 5.4kb 0 true false 8.6.2 true
unknown 5 r x.x.x.x _6ka 8506 3533 2012 60mb 19502 false true 8.6.2 false
unknown 5 r x.x.x.x _6kl 8517 2074 2075 42.1mb 19446 false true 8.6.2 false
unknown 5 r x.x.x.x _6ms 8596 618 1452 19.2mb 22340 false true 8.6.2 true
unknown 5 r x.x.x.x _6nm 8626 979 8774 75mb 21764 false true 8.6.2 false
unknown 5 r x.x.x.x _6nw 8636 49 457 3mb 54612 false true 8.6.2 true
unknown 5 r x.x.x.x _6o6 8646 57 830 6.1mb 57996 false true 8.6.2 true
unknown 5 r x.x.x.x _6og 8656 85 563 4.2mb 64276 false true 8.6.2 true
unknown 5 r x.x.x.x _6or 8667 89 1261 7.9mb 73404 false true 8.6.2 true
unknown 5 r x.x.x.x _6ot 8669 49 235 2.5mb 56148 false true 8.6.2 true
unknown 5 r x.x.x.x _6p0 8676 131 874 6.6mb 105780 false true 8.6.2 true
unknown 5 r x.x.x.x _6p9 8685 24 277 2.5mb 38892 false true 8.6.2 true
unknown 5 r x.x.x.x _6pb 8687 286 1388 13mb 204676 false true 8.6.2 true
unknown 5 r x.x.x.x _6pc 8688 68 165 1.9mb 69116 false true 8.6.2 true
unknown 5 r x.x.x.x _6pd 8689 0 1 32.6kb 10436 false true 8.6.2 true
unknown 5 r x.x.x.x _6pe 8690 33 94 940.9kb 41860 false true 8.6.2 true
unknown 5 r x.x.x.x _6pf 8691 49 112 1.2mb 50732 false true 8.6.2 trueIncreasing of k not helps
jmazanec15
commented
3 years ago @Hronom For search, the request will go to 6 shards. Routing will determine which shard (for example one search may go to the primary, the other may go to the replica).
Looking at shard 0, the replica has 38 segments, while the primary has 22 segments. During search, each segment will be searched sequentially and return k results. So, the hits for the primary is expected to be 30*22=660 while the hits for the replica would be 30*38=1140. Multiplying by 6 gives 3960 and 6840, respectively. The number of hits that you get are around 1854. It is so much fewer than 3960 and 6840 because many of the segments have less than or around 30 docs (so they aren't returning 30 results). However, this shows why when routing is not enabled the total hits are different. Increasing k should increase the total number of hits, regardless of whether routing is enabled or not. This is expected.
The post-filter will filter out those hits and return the top 3. To control the number of segments, you can periodically forcemerge. To increase the accuracy of the search per segment, you can increase m, ef_construction or ef_search.
However, before making any changes, I would first evaluate the recall with and without routing enabled. I do not think it should be impacted, but if it is, you can tweak those parameters above
We get different results depending on what shard request is landing. We have:
6shards,1replica100dimensions in vectorsR20210201-P2sizeset to3kset to30If we don't use
routingour results are jumping depending on what shard request is landing.The expected behavior: we get top k from each segment -> merge results from nodes by score on node that receives requests -> get top 3 from merged list -> top 3 goes in response.
Seems like a bug.
Please can you explain: what under the hood happens that leads to the different behavior?