timescale / pgvectorscale

A complement to pgvector for high performance, cost efficient vector search on large workloads.
PostgreSQL License
1.38k stars 58 forks source link

Requires block size to be 8K #92

Open levkk opened 5 months ago

levkk commented 5 months ago

Probably not a bug since running with block_size larger than 8K is not common, but this assert triggers on my setup (block_size = 32768):

assert!((*header).pd_special <= pgrx::pg_sys::BLCKSZ as u16);

Removing the assert causes cascading errors down the line:

ERROR:  assertion `left == right` failed
  left: 37060
 right: 44580

No idea where this assert is coming from, since backtrace isn't available.

cevian commented 5 months ago

what was the index definition and the number of dimensions?

levkk commented 5 months ago
=> \d+ embeddings_lev;
                                                           Table "public.embeddings_lev"
  Column   |    Type     | Collation | Nullable |                  Default                   | Storage  | Compression | Stats target | Description
-----------+-------------+-----------+----------+--------------------------------------------+----------+-------------+--------------+-------------
 id        | integer     |           | not null | nextval('embeddings_lev_id_seq'::regclass) | plain    |             |              |
 embedding | vector(384) |           |          |                                            | external |             |              |
Indexes:
    "embeddings_lev_pkey" PRIMARY KEY, btree (id)
Access method: heap
=> CREATE INDEX ON embeddings_lev USING diskann(embedding);

NOTICE:  Starting index build. num_neighbors=-1 search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression
ERROR:  assertion `left == right` failed
  left: 22672
 right: 44580
=> \dx
 vector                 | 0.6.0        | public     | vector data type and ivfflat and hnsw access methods
 vectorscale            | 0.2.0        | public     | pgvectorscale:  Advanced indexing for vector data

fwiw, running the index creation command multiple times produces different left/right assertion failures, e.g. :

ERROR:  assertion `left == right` failed
  left: 2427
 right: 44580