Dynamic Mapping

[inferno-umar]

Proposal:

Auto create column/s inside existing schema when new field is observed in incoming data!

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

- [ ] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [ ] Documentation reviewed - [ ] [Changelog](https://docs.google.com/spreadsheets/d/1mz_3dRWKs86FjRF7EIZUziUDK_2Hvhd97G0pLpxo05s/edit?pli=1&gid=1102439133#gid=1102439133) updated - [x] OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

Thanks for the feature request @inferno-umar

We could consider performing an implicit ALTER to add a new column when appropriate, for example internally it would look so:

mysql> drop table if exists t; insert into t(f) values(1.23); alter table t add column f2 int; insert into t(f2) values(123); alter table t add column f3 text; insert into t(f3) values('abc'); desc t; select * from t;
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t(f) values(1.23)
--------------

Query OK, 1 row affected (0.01 sec)

--------------
alter table t add column f2 int
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t(f2) values(123)
--------------

Query OK, 1 row affected (0.00 sec)

--------------
alter table t add column f3 text
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t(f3) values('abc')
--------------

Query OK, 1 row affected (0.00 sec)

--------------
desc t
--------------

+-------+--------+----------------+
| Field | Type   | Properties     |
+-------+--------+----------------+
| id    | bigint |                |
| f3    | text   | indexed stored |
| f     | float  |                |
| f2    | uint   |                |
+-------+--------+----------------+
4 rows in set (0.00 sec)

--------------
select * from t
--------------

+---------------------+------+----------+------+
| id                  | f3   | f        | f2   |
+---------------------+------+----------+------+
| 1516113395986727034 |      | 0.000000 |  123 |
| 1516113395986727035 | abc  | 0.000000 |    0 |
| 1516113395986727033 |      | 1.230000 |    0 |
+---------------------+------+----------+------+
3 rows in set (0.00 sec)
--- 3 out of 3 results in 0ms ---

[inferno-umar]

I was wondering the same thing...

Where should this implicit thing take place ? Inside the manticore-cpp code or the manticore-buddy interface...?

[sanikolaev]

Inside the manticore-cpp code or the manticore-buddy interface...?

Manticore Buddy. Somewhere here https://github.com/manticoresoftware/manticoresearch-buddy/tree/main/src/Plugin/Insert