Open MeltyBot opened 2 years ago
@aaronsteers I was thinking about something similar recently and stumbled on this issue. I think my idea is very similar but with more of a focus on installing configuration to enable features.
Problem:
Currently we have a good way to install, lock, and run plugins in a good/simple default way but configurations from there are left completely up to the user. This is for good reason and allows Meltano to be less opinionated and really flexible but some of the major DataOps benefits of using Meltano like isolated dev environments and CI are only enabled by Meltano and dont come out of the box. For example in the Squared project we show an opinionated way of how to do CI/isolated dev in Snowflake using Meltano. That functionality was enabled by Meltano but was really built/configured in the user's project on top of Meltano by leveraging the environments feature. That configuration was not trivial so the features are less accessible to users because they might not know how to configure it themselves. We should provide a way to share that configuration layer. Its opinionated so adding it to Meltano core doesnt feel like the right place. My thought is that Meltano should provide nice clean default unopinionated behavior to get plugins working then the ecosystem provides many variations of opinionated configuration layers on top for users to choose from.
Idea: Collections/Modules
I was thinking we'd have a "collections" or "modules" concept where someone could publish their set of their pre-configured plugins (collection.meltano.yml, lock files, instructions, etc.) and another user could add the whole set to their project. Maybe as a starting point but could also be added to an existing project. I wasnt thinking about having code included like suggested above with starva dbt models but I dont see why both couldnt be included. I originally started thinking about this as forkable projects or starter projects but thats less ideal to me because they require you to be starting from scratch.
Things I'm thinking about:
meltano add module snowflake-isolated-dev-envs
could be cool. Maybe its checks for plugin conflicts, installs files like plugin definitions if needed, dbt packages, and whatever else.This is kinda common in FE land, but Remix has a pretty neat "stacks" concept: https://remix.run/docs/en/v1/pages/stacks
Stacks cover things like what data stores and orm's are used and preconfigured, what css frameworks or utilities are included, etc. That means that someone like Netlify can also supply a stack: https://github.com/netlify-templates/kpop-stack.
For us, that might mean that we for example have a "hubspot bigquery stack" that sets you up with: tap-hubspot, target-bigquery, dbt-bigquery, and a GCP CloudRun based scheduler. Would get us a one step closer to a "one click fork and deploy" button.
A User highlighted this issue as a keen interest.
FYI, new comments on this topic are on the related issue:
My comment here, for instance: https://github.com/meltano/meltano/issues/6678#issuecomment-1228780222
Spec discussion:
Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/214
Originally created by @aaronsteers on 2022-03-14 11:55:36
We'd like community members to be able to submit Meltano data. This is analogous with "Data-Stack-in-a-Box", whcih users could download as their "starter project" or they could be added as subprojects within a larger data project.
Unlike dbt Hub packages and unlike Meltano's existing
transform
packages:meltano.yml
.Example use cases: