Support for local template files

benzvan commented 3 years ago

Description

The current template only supports github as the source type. Adding a file or local source would make testing templates locally much easier as well as making organizing vela files simpler.

templates:
  - name: local-template
    source: file://my/awesome/template.yml
    type: file

Value

Trying to validate template format locally, without pushing to a git branch, is currently impossible, or at least undocumented. (I couldn't see anything in the git library or in git's documentation to indicate a way to do it with git's format that worked.)

Definition of Done

Template files from the current repository can be used in a vela.yml file without fetching them from git.

Effort (Optional)

1-2 days

Impacted Personas (Optional)

Engineers writing and using templates will be able to save time and effort.

benzvan commented 3 years ago

If this is something the community agrees is useful, I'm happy to create a PR. I looked into the code while investigating this and have a good idea where/how to add the functionality. It will require a small refactor since the current code rejects any template that is not of type: github.

benzvan commented 3 years ago

I've forked this and started looking at it. Can't guarantee I'll finish it right away since we're concentrating on moving to vela right now and have a work-around.

JordanSussman commented 3 years ago

The current template only supports github as the source type. Adding a file or local source would make testing templates locally much easier as well as making organizing vela files simpler.

I'm thinking the experience for this could be somewhat frustrating since the user would still need to modify the .vela.yml. I'd propose some setup that simply utilized a feature flag within the CLI to tell it to look locally. For example, passing the flag of local-templates would attempt to resolve the templates from the local directory structure that matches the host.

benzvan commented 3 years ago

That could be useful for testing for sure and would solve my problem. What if someone actually wants to run a vela build with templates contained in the local repo though?

jbrockopp commented 3 years ago

@benzvan

That could be useful for testing for sure and would solve my problem. What if someone actually wants to run a vela build with templates contained in the local repo though?

If I'm following correctly, that should already be possible today 👍

Let's use this repo (go-vela/community) as an example.

And for this example, pretend that there is a template.yml file in the root of this repo.

I would be able to import that template directly into my Vela pipeline with the following:

version: "1"

templates:
  - name: local-template
    source: github.com/go-vela/community/template.yml
    type: github

steps:
<more_content>

The source parameter under templates roughly follows:

# syntax
<vcs_system>/<org>/<repo>/<path_to_template>

# example
git.company.com/myOrg/myRepo/path/to/template.yml

And you should be able to reference any template in the VCS system so long as the repo you're importing from is public.

JordanSussman commented 3 years ago

What if someone actually wants to run a vela build with templates contained in the local repo though?

Can you clarify what you mean by this scenario? I'm unclear if you are talking about running the build locally or within some deployed server.

benzvan commented 3 years ago

If I'm running a CI build in vela, then that template will be read from the default branch, not the branch of my PR.

So I have two scenarios: 1) I want to locally test templates with a vela-unittest framework I've just built and I'm needing to mock out the github response in order to fetch a local template file into my test-vela.yml file. Listing the template as a local file in that vela file would allow me to delete the mocked github code from my framework. 2) I want to (for some unknown reason) use local template files to keep my vela yml file cleaner and I want the CI build to use my branch's .vela.yml (which it does) and local template files (which it does not when using a github url)

JordanSussman commented 3 years ago

I want to (for some unknown reason) use local template files to keep my vela yml file cleaner and I want the CI build to use my branch's .vela.yml (which it does) and local template files (which it does not when using a github url)

I'd argue this should instead be resolved by allowing you to utilize variables within the 'templates' block to allow you to reference the current branch. Would that solution suffice?

benzvan commented 3 years ago

variables within the 'templates' block

That solves scenario two. I think it might be possible to use something like this to accomplish the same thing since the git library already supports branches.

source: github.com/org/repo/path/to/file.yml@{{BUILD_BRANCH}}

It doesn't solve any local actions where the code has not yet been pushed to a remote repo.

benzvan commented 3 years ago

Assuming you plan to support template types other than github, this change will be needed in the future anyway. (If you don't plan to support other template types, the Type field can be deleted.

Here's an example of an addition to the ExpandSteps function to accomplish the support:

        switch tmpl.Type {
        case "github":
            bytes, err = c.fetchGithubTemplate(tmpl)
        case "file":
            bytes, err = c.fetchFileTemplate(tmpl)
        default:
            logrus.Errorf("Unsupported template type: %v", tmpl.Type)
            continue
        }
        if err != nil {
            return yaml.StepSlice{}, yaml.SecretSlice{}, yaml.ServiceSlice{}, fmt.Errorf("invalid template source provided for %s: %v", step.Template.Name, err)

benzvan commented 3 years ago

Here's roughly what the whole thing would look like https://github.com/go-vela/compiler/pull/201

colindean commented 2 years ago

I'd like to see this implemented and extended to be a glob include of a directory of templates, each executed in parallel as a stage.

GitHub Actions works this way and I'm beginning to appreciate a significant benefit of this: atomic upsertion of workflow definitions. Right now, if I want to programmatically add a step or stage to a set of repos, I cannot safely parse YAML, modify it, and emit YAML, because there's no easy way to emit YAML in its source format without hooking into an AST. Inserting a workflow into a Starlark config necessitates skipping directly to the AST.

If I could include a directory in place or as a function to be called after some kind of from plugins import *, then I can simply push steps into a directory and let those be managed elsewhere.

My use case: My current team has about 4 base repos and another 7 app repos that look similar but must be separate codebases for deployment reasons at the moment. The app repo count grows at a rate of about one per quarter and we've sunset one created since we started. To make the same change to 11+ repos is laborious because of the manual work to change the .vela.yaml. Moving to Starlark might alleviate this, but it's be really nice to be able to use the GitHub API to upload a file and it's just… picked up. We'd add these files mostly for new checks like hadolint for Docker, codeowners-validator, Danger, etc. We could add these to some template that all of the repos consume, but we may not be wanting to add some component to all of them simultaneously. Having some kind of an opt-in switch takes us back to having to modify a bunch of repos, although it's a much smaller change.

kneal commented 2 years ago

Hey @colindean,

I think we probably should have re-named or closed this story after the PR implementation that wasn't exactly gonna hit the mark. The problem above after some 1on1s was really trying to solve local development with templates. That conversation really spawned this PR that's been around for a few releases. The goal was to enable more methods of testing templates locally. I can go in detail on why the above solution wouldn't work if you like but it might be better to just focus on your use case.

I'm not really sure how your use case is similar honestly. To me it sounds like the use case could be solved with templating out the steps but that misses on the opt-in component. Trying to read in-between the lines a little on the use case, but it sounds like you're more interested in the multiple file aspect GitHub Actions uses. So, essentially write a common workflow that a set of projects will always use and then upload that file to the projects that need the workflow. There was a multi-file issue somewhere but I'm failing to find it at the moment.

Another, interesting thing you said was Starlark, I'm not really sure how that solves the problem for you. Starlark is quite powerful and again I could be reading into this but was your thought that you'd write a module in Starlark in import it into Starlark pipelines? If, so that's not really a Vela thing, it's more we just need to update the compiler to allow Starlark vela files to support imports.

colindean commented 2 years ago

Trying to read in-between the lines a little on the use case, but it sounds like you're more interested in the multiple file aspect GitHub Actions uses. So, essentially write a common workflow that a set of projects will always use and then upload that file to the projects that need the workflow. There was a multi-file issue somewhere but I'm failing to find it at the moment.

Precisely. E.g. I can upload and manage a single file that handles a particular aspect without having to modify each consuming repo's Vela configuration.

Although, perhaps a template that calls templates is in order, but then all repos consuming that primary template would get the change and we'd need to opt into or opt out of secondary templates with some kind of feature flag. That could probably be done in an environment variable by pushing a secret, although the current state of the variable would not be visible since internal secrets are not readable by users, only running jobs. Could do it in an external secret, though. Either would be easily scriptable, and seemingly less intrusive since it's setting an envvar instead of committing, PRing, merging, and maintaining a file in the each repo. That's actually more approachable, now that I type it out 💡

My concept of how Vela consumes Starlark is underdeveloped because I have yet to use Starlark in a Vela config (for shame, Colin!) but yes: importing Starlark Vela files as a library like templates or like imports in Python could also address this in the same way I reasoned out above ☝️

So:

Glob inclusion of files in a directory as stages
Template of templates with feature flags set per-repo, in YAML
👆 but in Starlark

I think № 1 is what I came for and № 2 is probably what's available now with some finagling but № 3 is where I'd want to be, because my concept of Starlark is that its tooling for treating Starlark "code" as software is far more mature than gotemplate->yaml for validation, another thing I want to do to my Vela template dependencies.

cognifloyd commented 2 years ago

It looks like this issue can be closed since local template development has improved with https://github.com/go-vela/cli/pull/248

There are several tangents in here that could probably be their own feature requests, but keeping them in this issue is confusing and hard to grok.

template includes (source templates from the current repo on the same branch, not from an external git repo)
recursive templates: https://github.com/go-vela/community/issues/256
splitting the .vela.yml into multiple distinct pipelines: https://github.com/go-vela/community/issues/531

If I'm wrong and this should be re-opened, then please re-open. Or just file separate distinct issues and reference this one.

go-vela / community