Consider adding support for embeddings

[st-pasha]

• New parametric data type: float32[N], and possibly float16[N]. The data type is a plain array of N floats.
• Supported operations:
  • create from a plain list of python floats;
  • create from a list of N regular columns (which could also be numpy arrays);
  • dot-product;
  • cosine-similarity;
  • euclidian/manhattan distance;
  • jacard similarity;
  • others.

The LLMs are in fashion now, so why not add a support for them?

[oleksiyskononenko]

Can’t the existing array type be used for that?

[st-pasha]

You'd want an array of fixed size, kind of like a mathematical vector. It might be pretty similar to the existing array type in terms of implementation, though.

[oleksiyskononenko]

Yeah, but it probably could be array[float, N] type for fixed lengths vectors and just array[float] for arbitrary length.

[oleksiyskononenko]

I'm not an expert on LLMs, could you please point me to some information on the data types that they're using?

[st-pasha]

At the most basic level, an embedding is just a vector of floats of a fixed length. For example, see here: https://www.pinecone.io/learn/vector-database/