1.3 Normalize Data - Githubissues

raid-guild / gaianet-rag-api-pipeline

Supercharge your Gaianet node by generating a vector knowledge base from any API. Demo slides: https://hackmd.io/@santteegt/ByoykY4nC#/ Link to Docs below

https://raid-guild.github.io/gaianet-rag-api-pipeline/

MIT License

1 stars 0 forks source link

1.3 Normalize Data #7

Closed earth2travis closed 3 months ago

earth2travis commented 3 months ago

Normalize the extracted data to ensure consistency and compatibility with the target data schema. Normalization involves transforming the data into a standard format, addressing any discrepancies or variations in the raw data. This process is essential for maintaining data integrity and facilitating accurate analysis.

[X] Implement data normalization transform function based on data analysis from the partition stage

earth2travis commented 3 months ago

First version in place

santteegt commented 3 months ago

Need to re-evaluate the current strategy. Potential adjustments:

Concatenate long-form texts prior to partitioning
Eval OpenAPI spec to classify long-form texts and quantitative/qualitative metadata fields

santteegt commented 3 months ago

To consider in #22

Metadata as JSON or converted into a markdown table