search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.17k stars 985 forks source link

OpenMetadata Application Sepcification #15613

Open sushi30 opened 6 months ago

sushi30 commented 6 months ago

Refining the OpenMetadata App Specification

Quartz should not store any application data. It should all be populated from OM DB.

App Lifecycle

  1. init - when it is cold start
  2. run - for business logic
  3. close - not sure what goes here ??? I suggest 2 more stages: install. uninstall

App context

All application context will be provided from within OM (avoid getting config from external sources such as Quartz, ingestion framework, etc...)

  1. market place definition
  2. configuration specified in configuration file (example: SearchReIndex.yaml)
  3. user configuration (stored in the DB)

App Lifecycle

  1. install - Runs on app installation
  2. init() - Runs one time during the instantiation of job
  3. run() - executes everytime the job has run
  4. close() - executes every time the job has done
  5. shutdown() - Runs one time when the server is about to be shutdown
  6. uninstall - runs on app uninstall

Logging

  1. Each application will store logs in a store designated by OM.
  2. The store will be query-able by the UI in order to get details about a specific run.
  3. This is similar to what is done with the ingestion pipelines.

Example Implementation

Add a file appender for $HOME/logs/{appId}/latest.log.

Add an endpoint /api/v1/apps/{appId}/logs/latest that will return the contents of the file.

Tasks

harshach commented 6 months ago

@sushi30 we should also implement a log function to store the logs against a run. This should be queryable by UI to show the live logs of an application run cc @mohityadav766 similar to what we are doing in ingestion pipelines

sushi30 commented 3 weeks ago

Good example for community app: https://github.com/open-metadata/OpenMetadata/issues/15600