lanterndata / lantern

PostgreSQL vector database extension for building AI applications
https://lantern.dev
GNU Affero General Public License v3.0
790 stars 57 forks source link

create index with dynamically allocated blockmaps #25

Closed davkhech closed 1 year ago

davkhech commented 1 year ago

In this PR changing the logic how we allocate blockmap pages first we allocate one blockmap page and then fill in data pages, when there's not enough space we allocate two more blockmap pages then again data pages, and so on the number of blockmap pages increase by power of two

here's a little schema how blocks look like

hdr page -> blockmap page 1 -> data page 1, data page 2 ... data page gk -> blockmap page 2, blockmap page 3 -> data page gk + 1, data page gk + 2, .... data page gk + gf -> blockmap page 4 ... blockmap page 7 -> data page .... and so on

Ngalstyan4 commented 1 year ago

Thanks!

32 is a continuation of this work and the feature will be merged from there.