Map subdirectories to a specific workflow without defining all of the directories as individual projects

[humpalu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I have to create a custom workflow because of keep production and development related terraform codes in the same repository. Here is my sample environment

terraform/development/d-team-a
terraform/development/d-team-b
terraform/production/p-team-a
terraform/production/p-team-b
atlantis.yaml

       .
       └── terraform
           ├── development
           │   └── d-team-a
           │   └── d-team-b
           |── production
           │   └── p-team-a
           │   └── p-team-b
           └── atlantis.yaml

The folders are dynamically changed by our development teams. Its not possible to always update the atlantis.yaml and add/remove the subfolders. I need a custom workflow because I have different Azure credentials for development and for prod (e.g. ARM_SUBSCRIPTION_ID ) With my following yaml config , it always want to plan the development or production folder

Isnt there any ${PATH_TO_CHANGED_SUBFOLDER} env variable what contains the changed file or path?

Reproduction Steps

run with the same config

Logs

Provide log files from Atlantis server

logs can be retrieved from the deployment or from atlantis comments by adding --debug such as atlantis plan --debug

Logs

``` running "/home/atlantis/.asdf/shims/terraform plan -input=false -refresh -out \"/atlantis-data/repos/bestsecret/csp/runatlantis/1/default/terraform/development/development-default.tfplan\"" in "/atlantis-data/repos/bestsecret/csp/runatlantis/1/default/terraform/development": exit status 1 ╷ │ Error: No configuration files │ │ Plan requires configuration to be present. Planning without a configuration │ would mark everything for destruction, which is normally not what is │ desired. If you would like to destroy everything, run plan with the │ -destroy option. Otherwise, create a Terraform configuration file (.tf │ file) and try again. ```

Environment details

If not already included, please provide the following:

• Atlantis version: latest
• Deployment method: ecs/eks/helm/tf module: helm
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version:
• Atlantis flags:

Atlantis server-side config file:

repoConfig: |
  ---
  repos:
  - id: gitlab.com/bestsecret/csp/runatlantis
    allow_custom_workflows: true
    allowed_overrides: [workflow]
    delete_source_branch_on_merge: true
    repo_locking: true

Repo atlantis.yaml file:

version: 3
projects:
- name: development
  dir: terraform/development
  workflow: devflow
  autoplan:
    enabled: true
    when_modified: ["**/*.tf"]
- name: production
  dir: terraform/production
  workflow: prodflow
  autoplan:
    enabled: true
    when_modified: ["**/*.tf"]
workflows:
  devflow:
    plan:
      steps:
      - env:
          name: ARM_SUBSCRIPTION_ID
          command: 'export ARM_SUBSCRIPTION_ID=${ARM_SUBSCRIPTION_ID_DEV}'
 #     - run: echo ${ARM_SUBSCRIPTION_ID} && terraform plan 
      - init
      - plan
      - apply
  prodflow:
    plan:
      steps:
      - env:
          name: ARM_SUBSCRIPTION_ID
          command: 'export ARM_SUBSCRIPTION_ID=${ARM_SUBSCRIPTION_ID_PROD}'
      - run: terraform plan ${PATH_TO_CHANGED_SUBFOLDER}
    apply:
      steps:
      - env:
          name: ARM_SUBSCRIPTION_ID
          command: 'export ARM_SUBSCRIPTION_ID=${ARM_SUBSCRIPTION_ID_PROD}'
      - run: terraform apply $PLANFILE

Any other information you can provide about the environment/deployment (efs/nfs, aws/gcp, k8s/fargate, etc)

Additional Context

[pomcho555]

I had the same issue with the following file structure. I'm handling multi apps inside a single repository, Monorepo so to speak.

.
├── projectA
│   ├── prod
│   │   ├── appA
│   │   │   └── main.tf
│   │   └── appB
│   │       └── main.tf
│   └── stg
│       ├── appA
│       │   └── main.tf
│       └── appB
│           └── main.tf
├── projectB
│   ├── prod
│   │   ├── appA
│   │   │   └── main.tf
│   │   └── appB
│   │       └── main.tf
│   └── stg
│       └── appB
│           └── main.tf
└── sandbox
    └── appA
        └── main.tf

atlantis plan works without declaring projects stanza inside atlantis.yaml like this.

version: 3
automerge: true
delete_source_branch_on_merge: true
parallel_plan: true
parallel_apply: true

However once I add whatever projects definition under projects section, Atlantis bot says the below error because it didn't find the terraform files.

Error: No configuration files.

version: 3
automerge: true
delete_source_branch_on_merge: true
parallel_plan: true
parallel_apply: true
- name: whatever-project
  dir: .
  workspace: default

NOTE: It also doesn't work even if I specified ./**/*.tf as when_modified.

[humpalu]

Hello @pomcho555 , Correct. It works like that but unfortunately I have to make custom workflow per folder due some env variables should be different based on the folder structure. I need different SUBSCRIPTION_ID env variable for development and different for production.

[pomcho555]

According to the code, Atlantis has the capability to determine the appropriate folder for terraform plan, even for deeply nested sub-folders. However, when setting projects, it does not search for subfolders below the specified project folder. This behavior is a feature and not a bug. To address this limitation, you can define all subdirectories that you want to apply terraform plan to or consider implementing a new feature.

By the way, I would like to clarify that this bug is not specific to a particular cloud provider, as I am using AWS rather than Azure.

[humpalu]

In this case my problem is the "WHY?" It would be great if I can simply hit the

version: 3
projects:
- name: development
  dir: terraform/development/**
  workflow: devflow
  autoplan:
    enabled: true
    when_modified: ["**/*.tf"]

And it will run a custom workflow called "devflow" against of the changed files in the defined project directory? So the behaviour shall be exactly the same like the non-defined projects , only with an extra condition to use the defined workflow.

And yes, its not realted to any cloud provider. Its simply not working :-)

[pomcho555]

Because you cannot use regex or wildcard in dir section. I tried a similar dir specification and then log spit out the following error. It's not that smart as you expected.

"project at dir \"terraform/development/**\" not included because dir does not exist"

[nitrocode]

Related

• https://github.com/runatlantis/atlantis/issues/2708

• 3038

[jlestrada]

This also appears to be impacting our particular scenario where we wish to enable policy checking in a mono repo for only specific projects.

This may be a blocker for us to proceed with policy checking through Atlantis.

Is there any anticipation of addressing this issue?

Happy to try and look at the code as well if there is agreement that this is an unintended behavior or limitation. Just don't want to spend time if seen as non-issue or unwanted pattern

[jamengual]

PRs are always welcome, as mentioned before this is not a bug, it is working as intended. A new Feature could be added or the current one enhanced to allow regexes. this has been done at the project and dir level but not at the subfolder level.

[jlestrada]

Yeah i think that makes sense to me. If we are capable of enabling regex support for the attribute dir then I believe we resolve this issue.

This feature would also make it possible if i am not mistaken to enable auto plan for the repo by default without need to specify each project spacing. Seems like a good feature enhancement.

Thanks for the confirmation!

[nitrocode]

This is a duplicate of https://github.com/runatlantis/atlantis/issues/2708. Let's move discussion to the original issue.

[joedenniss]

The solution implemented by https://github.com/runatlantis/atlantis/pull/3895 which resolves https://github.com/runatlantis/atlantis/issues/2708 doesn't appear to solve the problem originally described here.

The solution in https://github.com/runatlantis/atlantis/pull/3895 enables a mixture of auto-discovered projects and statically defined ones. But the usecase here is to be able to map all the projects within a directory to a particular Atlantis workflow, without having to statically define all the individual projects.

[joedenniss]

Thought I'd share our workaround for anyone else with similar requirements. Our usecase is mapping top-level directories to AWS profiles which is similar to the OPs. We ultimately achieved this by removing the projects configuration entirely and creating a single custom workflow which sets the AWS_PROFILE variable dynamically based on the directory being planned (available via the REPO_REL_DIR variable provided by Atlantis).

repos:
  - id: "/.*/"
    workflow: custom
workflows:
  custom:
    plan:
      steps:
        - env:
            name: AWS_PROFILE
            command: "echo $REPO_REL_DIR | cut -d'/' -f1"
        - init
        - plan

In our case the top-level directory names just happen to correspond to our AWS profile names, but I suspect this could be extended to perform some sort of mapping of directories to variable values. For example you could place some sort of custom config file in each directory and change the command to extract the variable from that file.