manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
9k stars 503 forks source link

alter float vector broken #2543

Open tomatolog opened 1 month ago

tomatolog commented 1 month ago

Bug Description:

if create index without float vector then add in later with ALTER daemon failed to save disk chunk or create invalid index

create table test ( title text );
insert into test values ( 1, 'yellow bag' ), ( 2, 'white bag' );
alter table test add column image_vector float_vector knn_type='hnsw' knn_dims='4' hnsw_similarity='cosine';
insert into test values ( 10, 'yellow bag', (0.653448,0.192478,0.017971,0.339821) ), ( 20, 'white bag', (-0.148894,0.748278,0.091892,-0.095406) );
flush ramchunk test;

produces error

sphinxql-55> flush ramchunk test;
ERROR 1064: table 'test': FLUSH RAMCHUNK failed; TABLE UNUSABLE (HNSW error: data has 0 values, index 'image_vector' needs 4 values)

if repeat same steps but flush data prior to alter

create table test ( title text );
insert into test values ( 1, 'yellow bag' ), ( 2, 'white bag' );
flush ramchunk test;
alter table test add column image_vector float_vector knn_type='hnsw' knn_dims='4' hnsw_similarity='cosine';
insert into test values ( 10, 'yellow bag', (0.653448,0.192478,0.017971,0.339821) ), ( 20, 'white bag', (-0.148894,0.748278,0.091892,-0.095406) );
flush ramchunk test;

all got saved well but disk chunk got saved without knn file and disk chunk 1 saved with knn. Need to check how optimize merges these disk chunks.

Manticore Search Version:

6.3.6

Operating System Version:

linux

Have you tried the latest development version?

None

Internal Checklist:

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

- [ ] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [ ] Documentation reviewed - [ ] [Changelog](https://docs.google.com/spreadsheets/d/1mz_3dRWKs86FjRF7EIZUziUDK_2Hvhd97G0pLpxo05s/edit?pli=1&gid=1102439133) updated
tomatolog commented 1 month ago

seems related to https://github.com/manticoresoftware/manticoresearch/issues/2544