Previous PR ignored vector index modifications during VACUUM operations. This fixed issue with VACUUMing only for tables without ROWID because for tables with ROWID vector index will preserve old identifiers while rowid in base table after vacuum can change:
libsql> CREATE TABLE t_vacuum ( emb FLOAT32(2) );
libsql> INSERT INTO t_vacuum VALUES (vector('[1,2]')), (vector('[3,4]')), (vector('[5,6]'));
libsql> CREATE INDEX t_vacuum_idx ON t_vacuum(libsql_vector_idx(emb));
libsql> DELETE FROM t_vacuum WHERE rowid = 2;
libsql> SELECT rowid FROM t_vacuum;
1
3
libsql> SELECT rowid FROM t_vacuum_idx_shadow;
1
3
libsql> VACUUM;
libsql> SELECT rowid FROM t_vacuum;
1
2
libsql> SELECT rowid FROM t_vacuum_idx_shadow;
1
3
This PR changes the way we handle VACUUM - now we explicitly filter vector index shadow tables from VACUUM target tables because they will be consistently populated during inserts in the base table
libsql> CREATE TABLE t_vacuum ( emb FLOAT32(2) );
libsql> INSERT INTO t_vacuum VALUES (vector('[1,2]')), (vector('[3,4]')), (vector('[5,6]'));
libsql> CREATE INDEX t_vacuum_idx ON t_vacuum(libsql_vector_idx(emb));
libsql> DELETE FROM t_vacuum WHERE rowid = 2;
libsql> SELECT rowid FROM t_vacuum;
1
3
libsql> SELECT rowid FROM t_vacuum_idx_shadow;
1
3
libsql> VACUUM;
libsql> SELECT rowid FROM t_vacuum;
1
2
libsql> SELECT rowid FROM t_vacuum_idx_shadow;
1
2
Context
Follow up after PR https://github.com/tursodatabase/libsql/pull/1591
Previous PR ignored vector index modifications during VACUUM operations. This fixed issue with VACUUMing only for tables without ROWID because for tables with ROWID vector index will preserve old identifiers while rowid in base table after vacuum can change:
This PR changes the way we handle VACUUM - now we explicitly filter vector index shadow tables from VACUUM target tables because they will be consistently populated during inserts in the base table