Jobs should support a post-checkout-steps parameter

[Peter-Darton-i2]

Describe Request:

Add a post-checkout-steps parameter to all this orb's jobs to allow the caller to execute arbitrary additional steps after the checkout command.

At present, this orb can only build orbs where the orb's source is exactly equal to the code in source control - it can't support any "pre-orb-build compilation" etc. The addition of a "post-checkout-steps" parameter (defaulting to no steps) that's run after the "checkout" step would provide an extension point to allow callers to e.g. generate a command's script, or do a search/replace on existing code etc, in order to ensure that when the remainder of the orb build runs, it runs with the desired orb source instead of only what came directly from source control.

Examples:

Example of an orb build where a simple script is generated during a code-generation stage that is then available for use by the commands etc in the orb code.

config.yml:

version: 2.1
setup: true
orbs:
  orb-tools: circleci/orb-tools@11
  shellcheck: circleci/shellcheck@3

filters: &filters
  tags:
    only: /.*/

executors:
  small:
    docker:
      - image: cimg/base:stable
    resource_class: small

jobs:
  code-generation:
    executor: small
    steps:
      - checkout
      - run:
          command: |-
            ./generate_myscript.sh > /tmp/myscript.sh
            chmod a+x /tmp/myscript.sh
      - persist_to_workspace:
          root: /tmp/
          paths:
            - "myscript.sh"
workflows:
  lint-pack:
    jobs:
      - code-generation
      - orb-tools/lint:
          requires:
            - code-generation
          post-checkout-steps:
            - attach_workspace:
                at: src/scripts/
          filters: *filters
      - orb-tools/pack:
          requires:
            - code-generation
          post-checkout-steps:
            - attach_workspace:
                at: src/scripts/
          filters: *filters
      - orb-tools/review:
          requires:
            - code-generation
          post-checkout-steps:
            - attach_workspace:
                at: src/scripts/
          filters: *filters
      - shellcheck/check:
          requires:
            - code-generation
          post-checkout-steps:
            - attach_workspace:
                at: src/scripts/
          filters: *filters
      - orb-tools/publish:
          vcs-type: << pipeline.project.type >>
          requires:
            - orb-tools/lint
            - orb-tools/review
            - orb-tools/pack
            - shellcheck/check
          orb-name: myorg/myorbname
          filters: *filters
      # Triggers the next workflow in the Orb Development Kit.
      - orb-tools/continue:
          pipeline-number: << pipeline.number >>
          vcs-type: << pipeline.project.type >>
          requires: orb-tools/publish
          filters: *filters

and an orb command that uses that script:

description: Foo
steps:
  - run:
      name: Bar
      command: <<include(scripts/myscript.sh)>>

(and, in this example, the source code in github does not contain the src/scripts/myscript.sh file, it merely provides the ability to create it by running generate_myscript.sh).

Supporting Documentation Links:

Orb jobs already support an implicit pre-steps set of steps but these get executed before anything else in the orb's job, and this limits what they can do. e.g. as most jobs need to do a "checkout" step, and checkout requires an empty target directory, the pre-steps can't put anything in the directory that checkout will write to. This means that it's (currently) not possible to supplement/generate/alter the orb source code so that it's any different from what's contained directly in source control. i.e. orb-tools can test & package an orb whose source is contained in version control, but it doesn't support any usecase where the orb source requires any computation/processing prior to that test/packaging.

I've seen other orb jobs support an explicit "extra steps" parameter precisely to cater for this "users might want to do something the orb authors didn't anticipate" scenario; the orb-tools orb's jobs should follow this pattern.

e.g. every job within orb-tools (at least, every one that does a "checkout" step) should add a post-checkout-steps parameter (of type "steps") and run that parameters' steps immediately after the checkout.

...
parameters:
...
  post-checkout-steps:
    description: Extension point to allow additional steps to execute after the code checkout step.
    type: steps
    default: []
...
steps:
  - checkout
  - steps: <<parameters.post-checkout-steps>>
...

... and, "for completeness", I'd suggest that the "publish" job (which doesn't "checkout"; it's first step is an "attach_workspace") should get a post-attach-steps parameter instead.

[cybik]

An alternative way to run post-checkout steps would be the ability to disable checkout and run presteps before a job.

Though, for the sake of good design, I will already say that this would not solve the issue for jobs with build logic that requires checkout to be run after a number of prerequisite steps.

[Peter-Darton-i2]

You know, I think you're on to something there... 😁

The ability to disable checkout would 100% unblock my usecase (where I wanted some build-time generated source). ...and I believe that your worries about it not solving some jobs is unfounded: if the caller required checkout to be run after some prerequisite steps, they could still do that by including a checkout step within their presteps at a position of their own choosing.

I think this might be a better option; it'd effectively allow users to "replace" the checkout step with whatever functionality they require. e.g. in my original example (in the main description), I've defined a code-generation job that creates some files to add to the code under source control, but this new suggestion would make it possible to have a code-generation job that completely replaces the source and e.g. put all the desired orb source into a persist_to_workspace src/... folder and then effectively replace all the orb build jobs' checkout steps with a call to attach_workspace instead.

i.e. just making the checkout optional enables everything I was after, and more.

Comparing the two approaches... I think that adding a post-checkout-steps parameter is probably easier to implement and would result in fewer changes to the code ... but adding a boolean do-checkout parameter (defaulting to true) might be simpler to document. (but I wouldn't recommend we do both)

Hmm... if we take this "be able to replace the checkout step" idea further, I wonder if it'd be possible to make the checkout step itself be a parameterised set of steps which defaults to just a single call to "checkout"? Can CircleCI support non-empty defaults for a parameter of type "steps"? e.g. have a "checkout-steps" parameter that defaults to just calling "checkout" but allowing the caller to replace it with steps of their own choosing (which may or may not include a checkout)?

[KyleTryon]

Hey folks, I am a bit late here but I think the option to remove the checkout step may be exactly what you want (as you both have mentioned).

Here is what I think we could do:

1. Add a new parameter to the orb jobs simply called checkout as a boolean set to true by default.
2. If you wish to replace the checkout step, you set checkout to false, and add your own custom "checkout" command. Here is an example of the potential config:

orb-tools/pack:
  checkout: false
  pre-steps:
    - my-checkout

my-checkout can be defined at the top of your config file:

commands:
  my-checkout:
    steps:
      - run: <some code>
      - checkout

@Peter-Darton-i2 as for if you can "override" a default step in CircleCI, you can not unfortunately, but we can build out our own commands and such. This orb used to offer commands as well as jobs, but it was too bloated and confusing for people. People quite often did not understand the difference between the two. So this orb does aim to be a bit opinionated for the sake of user experience.