Proposed Alias Additions to Task Yaml

[estebanreyl]

Task Yaml alias additions

We have been discussing how to facilitate and improve on existing task yaml capabilities. In particular by looking at declaring aliases within the yaml files for reusability and general convenience. In current consideration we want something providing functionality similar to C's #define preprocessor directive, with some added changes, in particular:

Through a global alias OSS file
- More succinctly express common patterns. Eg. using $commit instead of {{.Run.Commit}}
- Provide aliases for pre-defined well known tools. Eg. using $curl instead of mcr.microsoft.com/acr-task-commands/curl:latest
    - Containers would be pulled if not present locally on execution.
    - This would also allow easier integration with acr's other acr tools like Gil's new prune feature
For this in particular we could allow distinct global definition files for users to choose from 
outside of the default 

# Using local files 
- Provide definition only files (for reusability) to be included in task yaml files in a similar
    fashion to including an import.

# Using local aliases
- Users may define new values or override global ones in the definition section.

Defined values would be accessed using the $ character as a preceding char. $$ used to escape $. All alias could also be disabled in the define portion of the yaml or the alias identifier $ can be re defined.

Alias definition should be hierarchical, such that the last override of an alias will be the final considered value, that is hierarchically speaking: local definitions > file definitions > Global definitions

As an example we could see:

Example global definitions:

    registry : {{.Run.Registry}}
    commit : {{.Run.Commit}}
    id : {{.Run.ID}}

Example task yaml file:

version: v1.0.0
    alias-src: # Import common definitions from external files (Read top down, further down indicates higher precedence)
        - org-defaults.yaml #Example of an organization default
        - proj-defaults.yaml #Additional definitions used by other local projects
    alias: # Local definitions (Take precedent over those from provided files) 
        - helm: cmd.azurecr.io/helm:v2.11.0-rc.2
    steps:
    # build website and func-test images, concurrently
    - build: -t $registry/hello-world:$id -f hello-world.dockerfile .
        when: ["-"]
    - build: -t hello-world-test  -f hello-world.dockerfile  .
        when: ["-"]
    # run built images to be tested
    - id: hello-world
        cmd: $registry/hello-world:$id
    - id: func-tests
        cmd: hello-world-test
        env:
        - test-target=hello-world
    # push hello-world if func-tests are successful  
    - push: 
        - $registry/hello-world:$id
    # helm deploy the updated hello-world image
    - cmd: >
        $helm update helloworld ./release/helm/ 
        --reuse-values 
        --set helloworld.image=$registry/hello-world:$id

Example org-defaults.yaml:

    singularity: mcr.microsoft.com/acr-task-commands/singularity-builder:3.3
    pack: 'mcr.microsoft.com/azure-task-commands/buildpack:latest pack'

[northtyphoon]

Does the imported file in alias-src need to be local file? Can I do something like https://github.com/azure/acr-tasks/alias.yaml?

[estebanreyl]

@northtyphoon I don't see why not? additionally, it would be interesting to see if there are some public ones people are using. Ideally we can build it with support for both local and remote files.

[shahzzzam]

@estebanreyl What if the alias file is in a private repo?

[estebanreyl]

@shahzzzam … Good question. We could try adding some form of support maybe for GitHub auth but I don't really see a good way to add general auth for such a purpose. Personally I believe given such a requirement local files are not unreasonable as users would likely be able to include these along with their also in all likelyhood private projects. Suggestions are welcome though, I am truthfully less familiar with what we can expect from users

[shahzzzam]

@estebanreyl hmm, if the context is remote and private, we do allow users to pass in the git access token during Task creation. @northtyphoon I believe if the alias.yaml file is in the same repo as the context, it will already be locally available when running the Task (because we download the context before running the task). But if it is in a different private repo, do we want users to send additional git access tokens for the task?

[estebanreyl]

@shahzzzam That means it would thus make sense to give support for at least that context? It gets hazier in terms of expected use for me later on allowing additional repos with access tokens.

[SteveLasker]

Confirming we're removing the need to specify $ in cmd references

# helm deploy the updated hello-world image
    - cmd: >
        helm update helloworld ./release/helm/ 
        --reuse-values 
        --set helloworld.image=$registry/hello-world:$id

[ehotinger]

@SteveLasker yes

[northtyphoon]

The feature has been merged https://github.com/Azure/acr-builder/pull/468