Incorrect total_found in "show meta;" for a "group X by". Can repeat for 1 index or distributed index across two servers.

[nuhusky001]

Bug Description:

GROUP 3 BY company_id limit 1 does not return the correct count. if we add max_matches at various values, the total_found keeps changing.

16 shards, on two servers. If I target one shard, vs the distributed set of 16 - same results.

Manticore Search Version:

6.3.4

Operating System Version:

Ubuntu 22.04.4 LTS

Have you tried the latest development version?

Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

- [ ] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [ ] Documentation reviewed - [ ] Changelog updated

[nuhusky001]

for 1 shard, I know there are 1064 total real results, but I can never get total_found to tell me this number. I get between 650-916 for total_found... but never 1064 (which is correct). if I put in " GROUP 3 BY company_id limit 2000 option max_matches=4000;" then I get the result set of 1064. But even in this case, I get total_results to be 916.

--- 1064 out of 916 results in 1s 724ms ---

Very rare - but sometimes I get a negative number for total_results with a "show meta;"

tried variable OPTION values - cutoff, max_matches, threshold, threads... all same results.

[sanikolaev]

Hi. Please show your full SHOW META output. Isn't the total_relation gte?

[nuhusky001]

its always "eq"

mysql> show meta; +----------------+-------------+ | Variable_name | Value | +----------------+-------------+ | total | 1000 | | total_found | 680 | | total_relation | eq | | time | 19.056 |

Could it be my config for building the plain indexes?

as you can see - it returned 1000 rows, but telling me it only found 680.

[nuhusky001]

GROUP 3 BY company_id limit 100 option max_matches=5000; +----------------+---------+ | Variable_name | Value | +----------------+---------+ | total | 100 | | total_found | 1967394 | | total_relation | eq | | time | 2.330 | | keyword[0] | us | | docs[0] | 3946746 | | hits[0] | 7589058 | +----------------+---------+

GROUP 3 BY company_id limit 100 option max_matches=100; +----------------+---------+ | Variable_name | Value | +----------------+---------+ | total | 100 | | total_found | 2534203 | | total_relation | eq | | time | 0.795 | | keyword[0] | us | | docs[0] | 3946746 | | hits[0] | 7589058 | +----------------+---------+

[sanikolaev]

@nuhusky001 would it be possible to get your data to reproduce the issue on our side?

[nuhusky001]

Yes - let me try to build a smaller plain index and see if it still breaks - how should I transmit it?

[nuhusky001]

I have a smaller index - 28,000 records. and I can reproduce the bug with a simple query. Let me know how to get your the files. (tar/gzip at 16MB)

mysql> SELECT company_id,employee_id FROM employee9bug WHERE MATCH('@(country_code)(US)') GROUP 3 BY company_id limit 10 option max_matches=1000; +------------+-------------+ | company_id | employee_id | +------------+-------------+ | 1445 | 3320 | | 1813 | 6216 | | 2056 | 7944 | | 3117 | 15720 | | 3515 | 18696 | | 3976 | 21784 | | 4340 | 24088 | | 4340 | 70792 | | 4499 | 25192 | | 4592 | 25816 | +------------+-------------+ 10 rows in set (0.03 sec) --- 10 out of 23151 results in 24ms ---

mysql> show meta; +----------------+-------+ | Variable_name | Value | +----------------+-------+ | total | 10 | | total_found | 23151 | | total_relation | eq | | time | 0.024 | | keyword[0] | us | | docs[0] | 28411 | | hits[0] | 54859 | +----------------+-------+ 7 rows in set (0.01 sec)

mysql> SELECT company_id,employee_id FROM employee9bug WHERE MATCH('@(country_code)(US)') GROUP 3 BY company_id limit 10 option max_matches=100; +------------+-------------+ | company_id | employee_id | +------------+-------------+ | 1445 | 3320 | | 1813 | 6216 | | 2056 | 7944 | | 3117 | 15720 | | 3515 | 18696 | | 3976 | 21784 | | 4340 | 24088 | | 4340 | 70792 | | 4499 | 25192 | | 4592 | 25816 | +------------+-------------+ 10 rows in set (0.06 sec) --- 10 out of 25758 results in 49ms ---

mysql> show meta; +----------------+-------+ | Variable_name | Value | +----------------+-------+ | total | 10 | | total_found | 25758 | | total_relation | eq | | time | 0.049 | | keyword[0] | us | | docs[0] | 28411 | | hits[0] | 54859 | +----------------+-------+ 7 rows in set (0.00 sec)

[sanikolaev]

Yes - let me try to build a smaller plain index and see if it still breaks - how should I transmit it?

The instruction is here https://manual.manticoresearch.com/Reporting_bugs#Uploading-your-data

[nuhusky001]

uploaded to manticore/write-only/issue-2488

2488.tar.gz with the plain index inside

[nuhusky001]

any update on this? Can you confirm you got the files?

[sanikolaev]

@nuhusky001 I confirm:

snikolaev@dev2:~/issue-2488/manticore_bug$ ls -la
total 33880
drwxr-xr-x 2 snikolaev snikolaev     4096 Aug  8 13:50 .
drwxrwxr-x 3 snikolaev snikolaev     4096 Aug 19 07:24 ..
-rwxr-xr-x 1 snikolaev snikolaev  3376760 Aug  8 13:50 employee_shrd9bug.spa
-rwxr-xr-x 1 snikolaev snikolaev  1771049 Aug  8 13:50 employee_shrd9bug.spb
-rwxr-xr-x 1 snikolaev snikolaev 16281711 Aug  8 13:50 employee_shrd9bug.spd
-rwxr-xr-x 1 snikolaev snikolaev   362081 Aug  8 13:50 employee_shrd9bug.spe
-rwxr-xr-x 1 snikolaev snikolaev     3354 Aug  8 13:50 employee_shrd9bug.sph
-rwxr-xr-x 1 snikolaev snikolaev  1286505 Aug  8 13:50 employee_shrd9bug.sphi
-rwxr-xr-x 1 snikolaev snikolaev  1625229 Aug  8 13:50 employee_shrd9bug.spi
-rwxr-xr-x 1 snikolaev snikolaev        0 Aug  8 13:50 employee_shrd9bug.spl
-rwxr-xr-x 1 snikolaev snikolaev     3584 Aug  8 13:50 employee_shrd9bug.spm
-rwxr-xr-x 1 snikolaev snikolaev  9793145 Aug  8 13:50 employee_shrd9bug.spp
-rwxr-xr-x 1 snikolaev snikolaev   160907 Aug  8 13:50 employee_shrd9bug.spt

and the bug with total_relation = eq where it should be not eq:

snikolaev@dev2:~/issue-2488/manticore_bug$ mysql -P9315 -h0

mysql> SELECT company_id, count(*) FROM employee9bug GROUP BY company_id limit 0 option max_matches=100; show meta;
Empty set (0.00 sec)
--- 0 out of 4493 results in 1ms ---

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 0     |
| total_found    | 4493  |
| total_relation | eq    |
| time           | 0.001 |
+----------------+-------+
4 rows in set (0.00 sec)

mysql> SELECT company_id, count(*) FROM employee9bug GROUP BY company_id limit 0 option max_matches=10; show meta;
Empty set (0.00 sec)
--- 0 out of 3646 results in 1ms ---

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 0     |
| total_found    | 3646  |
| total_relation | eq    |
| time           | 0.001 |
+----------------+-------+
4 rows in set (0.00 sec)