Address SQL API Inconsistency in Search and RAG Functionality

[bonesmoses]

The SQL function call for vectorize.table looks like this:

vectorize."table"(
    "table" TEXT,
    "columns" TEXT[],
    "job_name" TEXT,
    "primary_key" TEXT,
    "schema" TEXT DEFAULT 'public',
    "update_col" TEXT DEFAULT 'last_updated_at',
    "transformer" TEXT DEFAULT 'sentence-transformers/all-MiniLM-L6-v2',
    "index_dist_type" vectorize.IndexDist DEFAULT 'pgv_hnsw_cosine',
    "table_method" vectorize.TableMethod DEFAULT 'join',
    "schedule" TEXT DEFAULT '* * * * *'
) RETURNS TEXT

While vectorize.init_rag looks like this:

vectorize.init_rag(
    "agent_name" TEXT,
    "table_name" TEXT,
    "unique_record_id" TEXT,
    "column" TEXT,
    "schema" TEXT DEFAULT 'public',
    "transformer" TEXT DEFAULT 'tembo/meta-llama/Meta-Llama-3-8B-Instruct',
    "index_dist_type" vectorize.IndexDist DEFAULT 'pgv_hnsw_cosine',
    "table_method" vectorize.TableMethod DEFAULT 'append'
) RETURNS TEXT

Note the differences:

• agent_name rather than job_name
• table_name rather than table
• unique_record_id rather than primary_key
• column as a single TEXT rather than columns as a TEXT array.

Similar differences exist between the search and rag functions.

• agent_name rather than job_name

There's also the naming scheme inconsistency between the two methods in general, i.e. vectorize.table rather than vectorize.init_table.

Initial investigation suggests init_rag is essentially a wrapper for table. One potential solution is to deprecate init_rag entirely, as the rag function should work with any vectorized table with a corresponding project / agent / job name. Additionally, since the init_rag method came later and probably contains the "intended" nomenclature, back-porting this to the table call may be justified.

[ChuckHend]

Thanks @bonesmoses . I agree with assessment and the potential solution.

The only somewhat involved change I can think of would be is to make vectorize.rag() multi-column aware for retrieval. I think it would be acceptable to concatenate the columns used in vectorize.table(), since that is how it handles preparing the text before they are transformed to embedding. for example, columns => ARRAY['product_name', 'description'] would end up something like "map symbolic representation of a place" during a vectorize.rag() call on a multi-column vectorize.table(). These changes will need to be made somewhere near here.