MeltanoLabs / Meta

The why, what, and how of MeltanoLabs
MIT License
5 stars 1 forks source link

Implement dynamic versioning for 2 or 3 `MeltanoLabs`-managed connectors. #22

Closed aaronsteers closed 1 year ago

aaronsteers commented 2 years ago

One challenge in maintaining multiple connectors is the tension between:

  1. We want to release pinnable versions often so that users can pin be in the practice of pinning to version IDs and not pip install from main branch.
  2. Even in a fully automated release cycle, there's at least one PR and repo 'version bump' needed before the release can be published.

This approach would eliminate the second requirement, so we could in theory release new versions without needing to commit a version bump change back to the repo:

From the readme:

Poetry's typical version setting is still required in [tool.poetry], but you are encouraged to use version = "0.0.0" as a standard placeholder.

With the minimal configuration above, the plugin will automatically take effect when you run commands such as poetry build. It will update the version in pyproject.toml, then revert the change when the plugin deactivates. If you want to include a __version__ variable in your code, just put a placeholder in the appropriate file and configure the plugin to update it (see below) if it isn't one of the defaults. You are encouraged to use __version__ = "0.0.0" as a standard placeholder.

edgarrmondragon commented 2 years ago

@aaronsteers with dynamic versioning in place, would users be installing from a git tag instead of the default branch?

WillDaSilva commented 2 years ago

@edgarrmondragon If they installed from a branch instead of a tag, depending on how we configure the tool, they'd either get a dirty version (e.g. 1.2.3-dev0.acdf123), or the nearest tag upstream of the commit they installed from. Installing from a tag would be best, as is already the case imo except when doing rapid development with a new repository.

edgarrmondragon commented 2 years ago

@WillDaSilva ah, that's nice. So if I understand the use case correctly, once a user publishes a new release & tag from the GitHub UI a workflow is triggered with the following steps:

This seems like a good approach, and we could probably make it even easier for each repo by wrapping the workflow described above in a reusable action.

cc @aaronsteers

WillDaSilva commented 2 years ago

@edgarrmondragon Yup. It wouldn't surprise me if such a workflow was already available on the marketplace as an action. It has the benefit of running the workflow for the release/tag itself, so there's no risk of drift between the commit that updates the version, and the commit that gets built into the published wheel/images.

Note that this approach only works if building directly git already works, so it wouldn't, for instance, work for Meltano because we have to build the UI prior to building the wheel.

aaronsteers commented 1 year ago

@kgpayne - It occurs to me that tap-snowflake and target-snowflake would be good 'first repos' to try this approach with. What do you think of trying this there with those two repos, and we can use those as a learning opportunity?

visch commented 1 year ago

Certainly could try it, very low risk no one's using this right now! :D

aaronsteers commented 1 year ago

Closing as complete. For example implementations, see

Fantastic work, @kgpayne ! 🚀