Closed idiom-bytes closed 8 months ago
Although Pydantic is a great tool, it's not always necessary since we already have Polars data schemas that validate the data structure. Using Pydantic now would result in double validation.
The whole etl pipeline is now implemented using CSV + DuckDB.
It has been verified and merged into PR #685, which is a large Epic that includes many other updates to improve Lake/ETL/etc... https://github.com/oceanprotocol/pdr-backend/issues/685
Motivation
Update gql_data_factory to integrate PersistentStore and push data to raw tables.
Persist SLA Only new records should be fetched & inserted into db.
DoD: