orb-tools/pack <<include(...)>> directive treats absolute paths as local

Peter-Darton-i2 commented 1 year ago

Orb version:

11.6.1

What happened:

I've got a command (in src/commands/*.yml) which contains a run step:

steps:
  - run:
      command: <<include(/tmp/my_code.sh)>>

During the orb-tools/pack phase, it ran:

mkdir -p "$ORB_PARAM_OUTPUT_DIR" && circleci orb pack --skip-update-check "$ORB_PARAM_SOURCE_DIR" >"${ORB_PARAM_OUTPUT_DIR}orb.yml"

and this command failed with error

Error: An unexpected error occurred: could not open src/tmp/my_code.sh for inclusion

Exited with code exit status 255

Note: The file /tmp/my_code.sh does exist (it was pulled in with an attach_workspace pre-step step), but only at the location specified in the command definition (/tmp/my_code.sh), not at src/tmp/my_code.sh).

e.g. see build failure at https://app.circleci.com/pipelines/github/i2group/circleci-orb-node/59/workflows/a0bc10df-2fbe-41b0-8cdf-a32cb02ba477/jobs/613

Expected behavior:

The orb-tools pack code should've successfully included /tmp/my_code.sh instead of erroring and complaining about a file path that it wasn't asked to include.

i.e. Absolute paths should be treated as absolute paths rather than being assumed to be relative paths; only relative paths should be treated as relative to the ORB_PARAM_SOURCE_DIR directory.

Additional Information:

Note: It's not possible to use attach_workspace to place the code within the src directory because pre-step steps happen before the checkout step, and the checkout step fails unless the working area is empty (e.g. if someone had used attach_workspace to put my_code.sh into src/...). So, for checkout to work after an attach_workspace, the workspace file(s) need to be placed outside of the src folder.

KyleTryon commented 1 year ago

Hey there, Just to clarify, the include syntax isn't actually part of this orb. Instead, it's a command used in the CircleCI CLI for this orb. You can find more details in the documentation here: https://circleci.com/docs/orb-concepts/#file-include-syntax and the source code here: https://github.com/CircleCI-Public/circleci-cli/blob/d710994d37524e1fda4031dea4a0fc505e9704fa/process/process.go#L33.

At the moment, it's designed to include scripts from the orb's source. It's unlikely we'll change this now because it would be a breaking change to all existing orbs. As a workaround, you can move your files into the orb's source directory before packing.

If you feel strongly about this change, we'd love to hear your thoughts! Please open a new issue on the CLI repo here: https://github.com/CircleCI-Public/circleci-cli/issues/new/choose. Thanks for your understanding!

Peter-Darton-i2 commented 1 year ago

This was resolved by the release of orb-tools@12.

Prior to that, it wasn't possible to do any non-standard processing of the source because there were no "hook points" into its API; with orb-tools@12 you can tell it "don't do the checkout, let me do that" and do your own thing ... e.g. putting generated source code into the src folder.

CircleCI-Public / orb-tools-orb