Import `token_reduce.json` into Database and Support Variable Table Sizes

[p3nGu1nZz]

Is your feature request related to a problem? Please describe. Currently, our database implementation only supports fixed table sizes (384 columns). We need to import token_reduce.json into the database, which requires supporting variable table sizes based on the number of PCA components.

Describe the solution you'd like Create a new function in our database code that:

• Supports creating tables with a variable number of columns.
• Allows specifying the size of the table when creating it.
• Imports the reduced embeddings from token_reduce.json into the database.

Describe alternatives you've considered

• Modifying the existing implementation to support variable table sizes, but this might introduce complexity and potential issues with backward compatibility.
• Keeping the fixed table size and padding the reduced embeddings, but this is inefficient and not scalable.

Additional context This change is necessary to handle the reduced embeddings efficiently and flexibly. The new function should be able to:

• Read the token_reduce.json file.
• Create a table with the appropriate number of columns based on the PCA components.
• Insert the reduced embeddings into the table.

Example structure of token_reduce.json:

{
    "token1": [0.1, 0.2, 0.3],
    "token2": [0.4, 0.5, 0.6],
    "token3": [0.7, 0.8, 0.9],
    ...
}

[p3nGu1nZz]

we completed a majority of this issue, however when we build the token table its not populating. I suspect this is due to the different table sizes of our embedding from 384 to 3. i think the table size is hardcoded in many places which needs investigating.

the _opti file is generating correctly with our pca sklearn algorithm.