search

ministryofjustice / data-catalogue

Data catalogue • This repository is defined and managed in Terraform
MIT License
2 stars 0 forks source link

SPIKE: Consider Data Product API service design with respect to OpenMetadata's Ingestion API #96

Open tom-webber opened 11 months ago

tom-webber commented 11 months ago

User Story

As a developer I want to be assured that we are building a service with an extensible design
So that we can continue to develop at pace, and not get bogged down in implementation details or highly-coupled elements and layers.

Proposal

We need to understand if we should be building our ingestion service directly into OpenMetadata, or perhaps building a service that mirrors the ingestion framework of OpenMetadata. If we build our own service, we should consider using an Entity store to store the entities, entity relationships, and change events for our metadata.

Should we be focusing on building a service where the Serving Layer (API) is decoupled from the Storage Layer, to simplify our API?

The OpenMetadata platform has been built by data platform developers that have felt the pain-points we are yet to experience, so it would be wise to learn from their design and understand exactly what service we want to create.

Additional Information

OpenMetadata high-level design document (essential reading)

github-actions[bot] commented 9 months ago

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.