Improve ingest/ APIs to return a response that notifies dev if there's incoming data that does not conform to the DM schema
Keep a record of number of requests to ingest routes that contain bad data
Additional Context:
For data validation at the api level would be really nice so that in dev to know really quickly if the connector is producing data that doesn't conform to the schema, right now I have been looking at thousands of records manually + duckdb for high-level stats
ingest/
APIs to return a response that notifies dev if there's incoming data that does not conform to the DM schemaingest
routes that contain bad dataAdditional Context:
For data validation at the api level would be really nice so that in dev to know really quickly if the connector is producing data that doesn't conform to the schema, right now I have been looking at thousands of records manually + duckdb for high-level stats