PGVECTOR-ELIXIR extension: example with the classification of data using OpenAI

[ndrean]

1. An example of a usage of pgvector.

We want to classify a dataset using "a certain distance" among the rows. The idea is to send the data to the (rate limited) OpenAI api endpoint, get in return tokens for each row, store as a vector, and then apply a distance sorting query between rows based on this vector field.

Note: the distance operator <=> is also used with PostGIS.

2. pgvector and indexes

A summary of the post above, as found in the post!: pgvectorprovides a comprehensive, performant, and 100% open source database for vector data. As a developer, the key to getting performance from pgvector are:

• Ensure your query is using the indexes
• Ensure your indexes have the optimal list size
• Ensure you have enough memory for the index

[ndrean]

3. pgvector-elixir

I want to use pgvector-elixir with Elixir.

❌ I failed miserably to install the extension PGVECTOR with brew install pgvector because I installed Postgres.App, and make doesn't seem to work easily on my brand new M2.

🚀 I installed postgresql@14 via homebrew, then pgvector, and I can CREATE EXTENSION VECTOR;. It is just more annoying than using the Postgres.App.

[ndrean]

Next step, build a repo or Livebook to test this.

[ndrean]

A nice post on pgvector with ML in mind:

and pgvector and HNSW: