search

meltano / hub

The single source of truth for all Meltano plugins, including all available Singer Taps and Targets: https://hub.meltano.com
https://hub.meltano.com
50 stars 68 forks source link

Ability to register modular, fully-working Meltano projects/subprojects on the Hub #210

Open MeltyBot opened 2 years ago

MeltyBot commented 2 years ago

Spec discussion:

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/214

Originally created by @aaronsteers on 2022-03-14 11:55:36

We'd like community members to be able to submit Meltano data. This is analogous with "Data-Stack-in-a-Box", whcih users could download as their "starter project" or they could be added as subprojects within a larger data project.

Unlike dbt Hub packages and unlike Meltano's existing transform packages:

Example use cases:

MeltyBot commented 2 years ago

View 11 previous comments from the original issue on GitLab

pnadolny13 commented 2 years ago

@aaronsteers I was thinking about something similar recently and stumbled on this issue. I think my idea is very similar but with more of a focus on installing configuration to enable features.

Problem:

Currently we have a good way to install, lock, and run plugins in a good/simple default way but configurations from there are left completely up to the user. This is for good reason and allows Meltano to be less opinionated and really flexible but some of the major DataOps benefits of using Meltano like isolated dev environments and CI are only enabled by Meltano and dont come out of the box. For example in the Squared project we show an opinionated way of how to do CI/isolated dev in Snowflake using Meltano. That functionality was enabled by Meltano but was really built/configured in the user's project on top of Meltano by leveraging the environments feature. That configuration was not trivial so the features are less accessible to users because they might not know how to configure it themselves. We should provide a way to share that configuration layer. Its opinionated so adding it to Meltano core doesnt feel like the right place. My thought is that Meltano should provide nice clean default unopinionated behavior to get plugins working then the ecosystem provides many variations of opinionated configuration layers on top for users to choose from.

Idea: Collections/Modules

I was thinking we'd have a "collections" or "modules" concept where someone could publish their set of their pre-configured plugins (collection.meltano.yml, lock files, instructions, etc.) and another user could add the whole set to their project. Maybe as a starting point but could also be added to an existing project. I wasnt thinking about having code included like suggested above with starva dbt models but I dont see why both couldnt be included. I originally started thinking about this as forkable projects or starter projects but thats less ideal to me because they require you to be starting from scratch.

Things I'm thinking about:

pandemicsyn commented 2 years ago

This is kinda common in FE land, but Remix has a pretty neat "stacks" concept: https://remix.run/docs/en/v1/pages/stacks

Stacks cover things like what data stores and orm's are used and preconfigured, what css frameworks or utilities are included, etc. That means that someone like Netlify can also supply a stack: https://github.com/netlify-templates/kpop-stack.

For us, that might mean that we for example have a "hubspot bigquery stack" that sets you up with: tap-hubspot, target-bigquery, dbt-bigquery, and a GCP CloudRun based scheduler. Would get us a one step closer to a "one click fork and deploy" button.

tayloramurphy commented 2 years ago

A User highlighted this issue as a keen interest.

aaronsteers commented 2 years ago

FYI, new comments on this topic are on the related issue:

My comment here, for instance: https://github.com/meltano/meltano/issues/6678#issuecomment-1228780222