Terragrunt run-all plan failing with exec.ExitError exit status 126.

[nishilampally]

Recently we have implemented tfenv and tgenv to manage multiple versions of terraform and terragrunt . When I try to run terragrunt run-all plan I'm getting a 126 error exit code. I've enabled debug logging and it's not very helpful. The issue is intermittent where run--all plan get succeeds sometimes and fails intermittently. User(AzDevOps) has execute permissions on both terraform and terragrunt. I am not sure why I am getting this error. There is not much information available from the logs.

Setup: Ubuntu machine 20_04-lts

Command: terragrunt run-all plan --terragrunt-source-update -no-color --terragrunt-include-external-dependencies

Permissions:

drwxr-xr-x 3 root root 4096 Sep 20 16:48 aws-cli drwxr-xr-x 3 root root 4096 Sep 20 16:48 bin drwxr-xr-x 2 root root 4096 Aug 31 21:39 etc drwxr-xr-x 2 root root 4096 Aug 31 21:39 games drwxr-xr-x 2 root root 4096 Aug 31 21:39 include drwxr-xr-x 3 root root 4096 Aug 31 21:39 lib lrwxrwxrwx 1 root root 9 Aug 31 21:39 man -> share/man drwxr-xr-x 2 root root 4096 Aug 31 21:39 sbin drwxr-xr-x 5 root root 4096 Sep 20 16:44 share drwxr-xr-x 2 root root 4096 Aug 31 21:39 src drwxrwxrwx 10 root root 4096 Sep 20 16:46 tfenv drwxrwxrwx 9 root root 4096 Sep 20 16:46 tgenv /agent/_work/1/s total 60

total 44 -rwxrwxrwx 1 root root 304 Sep 20 16:46 CHANGELOG.md -rwxrwxrwx 1 root root 1086 Sep 20 16:46 LICENSE -rwxrwxrwx 1 root root 5440 Sep 20 16:46 README.md drwxrwxrwx 2 root root 4096 Sep 20 16:46 assets drwxrwxrwx 2 root root 4096 Sep 20 16:46 bin drwxrwxrwx 2 root root 4096 Sep 20 16:46 libexec -rwxrwxrwx 1 root root 1165 Sep 20 16:46 list_all_versions_offline drwxrwxrwx 2 root root 4096 Sep 20 16:46 test -rwxrwxrwx 1 root root 7 Sep 20 16:46 version drwxrwxrwx 4 root root 4096 Oct 19 19:41 versions /agent/_work/1/s total 8 drwxr-xr-x 2 AzDevOps AzDevOps 4096 Oct 19 19:42 1.0.1 drwxrwxrwx 2 root root 4096 Sep 20 16:46 1.5.5 total 8 drwxr-xr-x 2 AzDevOps AzDevOps 4096 Oct 19 19:41 0.37.1 drwxrwxrwx 2 root root 4096 Sep 20 16:46 0.49.1 /agent/_work/1/s total 78812 -rwxr-xr-x 1 AzDevOps AzDevOps 80702103 Jun 24 2021 terraform total 35700 -rwxr-xr-x 1 AzDevOps AzDevOps 36554016 Oct 19 19:41 terragrunt /agent/_work/1/s -rwxr-xr-x 1 AzDevOps AzDevOps 80702103 Jun 24 2021 /usr/local/tfenv/versions/1.0.1/terraform -rwxr-xr-x 1 AzDevOps AzDevOps 36554016 Oct 19 19:41 /usr/local/tgenv/versions/0.37.1/terragrunt

Error:

time=2023-11-06T03:03:51Z level=error msg=169 errors occurred:

• Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado_access_policies/awslandingzone_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/key_vault_ado]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault_access_policies/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault, /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• exit status 126
• exit status 126
• Cannot process module Module /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_postgres_database_azure_com (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/apps/dea/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/dea/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/dea/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/agent_pool/vmss_iac (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/prd/network/subnet_adoagents, /agent/_work/1/s/prd/apps/omnicc/agent_pool/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/network/subnet_adoagents (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/network/subnet_adoagents (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/resource_group_vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/app_group]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/aks (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/container_registry_forestersfinancial, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace, /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_canadacentral_azmk8s_io, /agent/_work/1/s/prd/network/subnet_kubernetes, /agent/_work/1/s/prd/network/route_table_kubernetes, /agent/_work/1/s/prd/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_share_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_canadacentral_azmk8s_io (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
• Cannot process module Module /agent/_work/1/s/shared_cac/network/subnet_azurefw (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/vnet]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/resource_group_vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126

[iokiwi]

Note that a similar issue has been raised for tgenv/tgenv here https://github.com/tgenv/tgenv/issues/27