Add pre-pack to flatten directory structures

[emmeowzing]

What

Allows you to group commands and jobs (or even deeper) in their own subdirectories together.

How

Adds a pre-pack step that condenses the directory structure into a list of files. So a directory structure like

$ tree -a src/
src/
├── commands
│   ├── context
│   │   ├── add-env-vars.yml
│   │   ├── create.yml
│   │   ├── delete-env-vars.yml
│   │   ├── delete.yml
│   │   ├── get.yml
│   │   ├── list-env-vars.yml
│   │   └── list.yml
│   ├── insights
│   │   └── summary-metrics.yml
...

can be reduced to

$ tree -a src/
src/
├── commands
│   ├── context
│   │   ├── add-env-vars.yml
│   │   ├── create.yml
│   │   ├── delete-env-vars.yml
│   │   ├── delete.yml
│   │   ├── get.yml
│   │   ├── list-env-vars.yml
│   │   └── list.yml
│   ├── context-add-env-vars.yml
│   ├── context-create.yml
│   ├── context-delete-env-vars.yml
│   ├── context-delete.yml
│   ├── context-get.yml
│   ├── context-list-env-vars.yml
│   ├── context-list.yml
│   ├── insights
│   │   └── summary-metrics.yml
│   ├── insights-summary-metrics.yml
...

Note that the files are now prefixed by their former containing directory's name.

I've also included a script to accomplish the reverse for any local shell users. I typically add the following to my package.json -

{
...
"scripts": {
    "install:pre-commit:mac": "brew install pre-commit",
    "install:pre-commit:pip": "pip install pre-commit",
    "install:pre-commit-hooks": "pre-commit install --install-hooks --allow-missing-config -t pre-commit -t prepare-commit-msg",
    "orb:pack": "find src -maxdepth 1 -mindepth 1 -type d | xargs -I % basename % | xargs -I % ./scripts/pre-pack.sh src % && circleci orb pack src/ > orb.yml",
    "orb:validate": "circleci orb validate orb.yml",
    "orb:cleanup": "find src -maxdepth 1 -mindepth 1 -type d | xargs -I % basename % | xargs -I % ./scripts/rev-pack.sh src %; yes | rm orb.yml",
    "orb:clean": "yarn orb:cleanup",
    "orb:cleanup:force": "find src -maxdepth 1 -mindepth 1 -type d | xargs -I % basename % | xargs -I % ./scripts/rev-pack.sh src % true; yes | rm orb.yml",
    "orb:clean:force": "yarn orb:cleanup:force"
  }
...
}

so I can call these steps more easily, and against every subdirectory under src/.

Why

As far as use-cases, I've found myself wrapping CLIs and APIs as of late, and the base directories under src/commands and src/jobs get crowded for complex interfaces. Plus, it's easier to navigate if you're able to group related commands (which I'd typically prefix with the directory name, anyway, so they're grouped together in my editor's directory list).

[emmeowzing]

Example config with prepacking - https://github.com/bjd2385/dynamic-continuation-orb/blob/master/.circleci/config.yml#L117

Note that it can be safely added/ran whether there are subdirectories to traverse, or not.

[KyleTryon]

Hello @bjd2385

Could you help me better understand the problem that is being resolved by this change? It doesn't appear as though your example takes advantage of the feature: https://github.com/bjd2385/dynamic-continuation-orb/tree/master/src/commands

[emmeowzing]

@KyleTryon here's a better example in an incomplete orb I just made public, also appears to be what I used above -

https://github.com/bjd2385/circleci-api-orb/tree/master/src/commands

And a screenshot demoing -

[emmeowzing]

Even if this is the wrong place to add it, it'd be cool if this were implemented in circleci orb pack, so maybe a feature request in circleci's forums.

[KyleTryon]

Thanks for that link @bjd2385,

So in short, would you say the goal is to further group elements within folders to add prefixes to their names? In this case you have several commands that start with context.

---

We typically have followed a paradigm of attempting to limit the number commands a user needs to use and instead trying to write "intelligent" commands when possible, using parameters, conditionals, and complex bash, as well as trying to offer jobs where possible. I wonder in this case, is it possible the number of commands available could be reduced, or should be reduced and could more parameters allow the user to write less config, and move more of the logic into the orb.

---

For the concept, im hesitant to move on this at the moment but I like the work you did here with Yarn and would like to allow some time for the community to chime in.

If we were to make a change like this though, as you mentioned, it may be better to add this to the orb pack command in the cli here: https://github.com/CircleCI-Public/circleci-cli

[emmeowzing]

So in short, would you say the goal is to further group elements within folders to add prefixes to their names? In this case you have several commands that start with context.

Yes, this is the case. The example orb is unfinished, but that's the goal - to group several related commands together in subdirectories, and the same for jobs, and so on. And they are logically reduced to a config by prefixing the file name with the subdirectory name within which they are contained.

I look forward to seeing what the community thinks, but I will also take a look at the circleci CLI repo as well to see what it would take to add this kind of functionality there.

[KyleTryon]

Going to close out this issue now due to lack of outside interest. I would recommend for the time-being for anyone who wishes to replicate this, to fork the orb to customize to your liking. There will be many different potential patterns people want to follow, but I think we will keep the official orb-tools-orb opinionated.

Though it isn't exactly ready for developing orbs, the Config SDK comes to mind as well as another possible solution here: https://github.com/CircleCI-Public/circleci-config-sdk-ts

If we would like to revisit this, please open a new issue referencing this one. Thank you