Add ability for setup steps (pre_publish) for e.g Vault integration

[tdeekens]

Describe Request:

This Orb's functionality are all provided using jobs. They run on their own executor and spin up a new environment.

Secrets for publishing or commenting on GitHub have to be present as environment variables. These have to be defined in a project or organisation context as otherwise they would be be unavailable in the publish job of this Orb.

Secrets are not always defined on CircleCI and can also be pulled from a secret management system such as Vault.

Currently, this Orb has no way to integrate with secrets (at least in a way I know) of such an external tool.

A solution would be to allow to define steps running before publishing. This could be called pre_publish. These steps would allow integration with eg Vault using the OIDC tokens and can expose secrets on the environment (BASH_ENV) to be later available.

Examples:

orb-tools/publish:
   - pre_publish:
       - run:
           command: |
              vault kv get...
              echo "" >>> $BASH_ENV

This would also allow to follow this pattern.

Supporting Documentation Links:

[Peter-Darton-i2]

IME this kind of thing can be achieved using a job's pre-steps built-in parameter - any steps specified there will be run before any other steps in the job (so make sure you don't write files into the current directory or any subsequent checkout will fail).

[tdeekens]

IME this kind of thing can be achieved using a job's pre-steps built-in parameter

Thanks for getting back @Peter-Darton-i2. Does every job have that build in even if it's not defined in a job's specification?

[Peter-Darton-i2]

Yup. It's documented somewhere in the CCI docs (but, weirdly, not in the main reference docs where you'd expect). Every job you mention in a workflow can have pre-steps and post-steps added to them. pre-steps get executed before any job steps happen. post-steps happen after all the other job steps (so you might need a when: always if you want those to happen even if the job failed).

The problem comes when a job's internal implementation needs the working directory to be empty (e.g. it does a checkout or attach_workspace: at: . within its own steps, e.g. the orb-tools' publish command does this) and you want your pre-steps to do stuff with files in the working directory. e.g. in my usecase, I wanted to generate some orb source code to supplement what was obtained from source control. For pre-steps to be useful in that situation, the job also needs some form of "skip the checkout" or "skip the workspace_attach" flag, which wasn't available in orb-tools v11 ... but which was added in orb-tools v12.

TL;DR: Yes, pre-steps and post-steps are something you can do on any job, but that doesn't mean that they're useful on every job in every situation ... but for what you're doing, they should work perfectly.

Suggestion: Put your vault kv get stuff into a command and then you can just have a pre-steps: [ get_vault_credentials ] line everywhere it's needed.

...just BEWARE that any secrets that you use which aren't passed in via a CircleCI context will not have any secrets-obfuscation applied, so if your code reveals the secret by e.g. logging it to the console, that'll be shown in cleartext for anyone (who has access to your CircleCI build) to read. There isn't (currently) any means of telling CircleCI "I've got this secret and I want you to help ensure I don't log that to the console"; that functionality is built-in to contexts but not available by any other means.

[tdeekens]

For the record, this works for us. Huge thank you and 🤗 to @Peter-Darton-i2 for the suggestion.