1.3 Normalize Data - Githubissues

raid-guild / gaianet-rag-api-pipeline

0 stars 0 forks source link

1.3 Normalize Data #7

Closed earth2travis closed 3 weeks ago

earth2travis commented 1 month ago

Normalize the extracted data to ensure consistency and compatibility with the target data schema. Normalization involves transforming the data into a standard format, addressing any discrepancies or variations in the raw data. This process is essential for maintaining data integrity and facilitating accurate analysis.

[X] Implement data normalization transform function based on data analysis from the partition stage

earth2travis commented 1 month ago

First version in place

santteegt commented 1 month ago

Need to re-evaluate the current strategy. Potential adjustments:

Concatenate long-form texts prior to partitioning
Eval OpenAPI spec to classify long-form texts and quantitative/qualitative metadata fields

santteegt commented 4 weeks ago

To consider in #22

Metadata as JSON or converted into a markdown table