Show Meta for Buddy requests

[donhardman]

Proposal:

We should consider a way to implement sharing meta-information for performed queries with the buddy by directly sending the meta-info in the response for the buddy, in case we detect the proper User Agent.

In that case, all queries that are proxied to the buddy and executed by it will be able to accumulate and aggregate this info, and then respond to the daemon using an internal protocol. This will help us introduce a "show meta" feature for buddy-executed queries.

The issue is that currently, the "show meta" feature gives us the wrong response and contains an error when we have a query that was processed by the buddy.

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

- [x] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [ ] Documentation reviewed - [x] Changelog updated - [x] OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/2235

[sanikolaev]

Blocked by https://github.com/manticoresoftware/manticoresearch/issues/2235

2235 is done.

[tomatolog]

need more information about the exactly what to do here?

performed queries with the buddy

What interface or endpoints daemon should check for

consider a way to implement sharing meta-information

What meta information daemon should share and how the reply from the daemon to buddy should looks like?

[donhardman]

This task is about adding implementation to display information for SHOW META after SELECT in the same way we did it here, but for logging: https://github.com/manticoresoftware/manticoresearch/issues/2235

We need to store the information that buddy sends to us and display it when the user calls SHOW META.

Feel free to message me in case questions

[tomatolog]

after https://github.com/manticoresoftware/manticoresearch/issues/2235 buddy replies to daemon with additional meta property

{result:[], meta:[]}

these meta info used for query log. And this ticket needs also to store this meta at the daemon session::meta structure to show user in case user asked for it via show meta.

As for now show meta after requests fixed by buddy is empty.

[tomatolog]

fixed at https://github.com/manticoresoftware/manticoresearch/commit/1b285c5d5b3c313a54cbb13654850eef155f3dc7 to updated session::meta from the buddy reply

[tomatolog]

set ticket to @PavelShilin89 to add CLT test. If @donhardman has no ideas for test cases could close the issue

[sanikolaev]

Quick demo of how it works:

mysql> drop table if exists t; create table t(f text) min_infix_len='2'; insert into t values(1, 'abcdef'); select * from t where match('abcdef'); show meta; select * from t where match('abcef') option fuzzy=1; show meta;
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.63 sec)

--------------
create table t(f text) min_infix_len='2'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t values(1, 'abcdef')
--------------

Query OK, 1 row affected (0.15 sec)

--------------
select * from t where match('abcdef')
--------------

+------+--------+
| id   | f      |
+------+--------+
|    1 | abcdef |
+------+--------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

--------------
show meta
--------------

+----------------+--------+
| Variable_name  | Value  |
+----------------+--------+
| total          | 1      |
| total_found    | 1      |
| total_relation | eq     |
| time           | 0.000  |
| keyword[0]     | abcdef |
| docs[0]        | 1      |
| hits[0]        | 1      |
+----------------+--------+
7 rows in set (0.00 sec)

--------------
select * from t where match('abcef') option fuzzy=1
--------------

+------+--------+
| id   | f      |
+------+--------+
|    1 | abcdef |
+------+--------+
1 row in set (0.01 sec)

--------------
show meta
--------------

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 1     |
| total_found    | 1     |
| total_relation | eq    |
| time           | 0.006 |
+----------------+-------+
4 rows in set (0.00 sec)

Query log:

/* Wed Nov 13 06:37:59.567 2024 conn 18784 (127.0.0.1:36514) real 0.000 wall 0.000 found 1 */ SELECT * FROM t WHERE MATCH('abcdef');
/* Wed Nov 13 06:37:59.574 2024 conn 18784 (127.0.0.1:36514) real 0.006 wall 0.006 found 1 */ select * from t where match('abcef') option fuzzy=1;

@PavelShilin89 pls improve your fuzzy search tests by adding there show meta a query log test and make sure all works fine. Don't forget to test edge cases (like e.g. https://github.com/manticoresoftware/manticoresearch-buddy/issues/395)