Open danthelion opened 2 years ago
@danthelion thats a great idea! Thanks for opening this issue.
I havent had a chance to use data-diff yet but it looks very cool from what I've been hearing.
The plugins that are discoverable by Meltano are sourced from the hub so a simple implementation could be to define the package, its settings, relevant commands, and options docs for the hub plugin page. Check out https://github.com/meltano/hub/blob/main/_data/meltano/utilities/datahub/datahub-project.yml which is a relatively recent utility that someone contributed. You can just open a PR with that plugin definition yaml and a logo image.
Other helpful context:
The current state is that utility plugins are assumed to be pretty simple so defining settings/commands is sufficient but we're finding that it would be helpful to enable plugin developers to build "glue code" for the utility (see the Python-based Plugin Architecture issue), things like pre and post hooks for prep and cleanup work or other things to simplify the integration between plugins in your meltano project/stack. A few good examples are Airflow and Superset which are implemented in Meltano core right now and do things like compile config files, init databases, etc. then clean up after themselves on exit.
Theres work slated for next week to start pulling Superset/Airflow's glue code out of Meltano core and into standalone python packages, starting to iron out a framework and paving the way for other plugins.
I'm giving the extra context in case you have some ideas while working with data-diff around simplifying the startup/integration work needed to use the tool that go beyond just settings/commands. Things like auto configuring data-diff based on the Meltano extractor/loader or something like that 🤷 . We should have a way to support those integrations soon!
https://github.com/datafold/data-diff
Would be pretty neat to quickly verify loader jobs. I would love to implement this I'm just not sure where to start with utility plugins. In
discovery.yml
there are entries pointing to GitLab repos that have been deprecated in favor of the GitHub version but those are not in this config file, what's the process of adding a utility like this?