VladRassokhin / intellij-hcl

HCL language support for IntelliJ platform based IDEs
Apache License 2.0
244 stars 48 forks source link

Cannot locate module locally #365

Open solarmosaic-kflorence opened 2 years ago

solarmosaic-kflorence commented 2 years ago

Prerequisites

Installation details

Terraform Configuration Files

module "iam" {
  source = "s3::https://s3.amazonaws.com/path/to/module.zip"
}

And here is the project folder (including .terraform cache):

.
├── .terraform
│   ├── modules
│   │   ├── iam
│   │   │   ├── Makefile
│   │   │   ├── main.tf
│   │   │   ├── modules
│   │   │   │   ├── iam
│   │   │   │   │   ├── .terraform.lock.hcl
│   │   │   │   │   ├── main.tf
│   │   │   │   │   ├── outputs.tf
│   │   │   │   │   └── variables.tf
│   │   │   │   ├── local-data
│   │   │   │   │   └── outputs.tf
│   │   │   │   └── path
│   │   │   │       ├── outputs.tf
│   │   │   │       └── variables.tf
│   │   │   ├── output.tf
│   │   │   ├── policies
│   │   │   │   └── assume-role
│   │   │   │       └── codebuild.json
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   ├── iam.path.path
│   │   │   ├── go.mod
│   │   │   ├── go.sum
│   │   │   ├── infra
│   │   │   ├── output.tf
│   │   │   ├── readme.md
│   │   │   └── variables.tf
│   │   ├── iam.session
│   │   │   ├── Makefile
│   │   │   ├── main.tf
│   │   │   ├── modules
│   │   │   │   └── local-data
│   │   │   │       └── outputs.tf
│   │   │   ├── outputs.tf
│   │   │   └── versions.tf
│   │   └── modules.json
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── aws
│                   └── 3.58.0
│                       └── darwin_amd64
│                           └── terraform-provider-aws_v3.58.0_x5
├── .terraform.lock.hcl
└── main.tf

Exception

N/A

Expected Behavior

The HCL plugin should be able to reference any module cached in the ./.terraform in the current directory. We are using Terraform inside of a monorepo, so we have several directories scattered around with Terraform configuration. I have noticed in some project folders we do not have this issue and in others we do. It seems to happen most often within sub-modules (e.g. ./modules/bar, relative to the project root), or when referencing sub-folders of a module, for example:

module "bar" {
  source = "s3::https://s3.amazonaws.com/path/to/module.zip//path/to/sub-module"
}

Actual Behavior

 Cannot locate module locally: Terraform Module 'iam' with source 's3::https://s3.amazonaws.com/path/to/module.zip' directory not found locally, use `terraform get` to fetch modules.

The module exists, of course, and is downloaded and cached in the .terraform folder. Performing Terraform CLI commands in the directory will succeed. It's just a problem with the plugin cache.

. <-- in this folder the module was resolved, and it also resolved it in the sub-folder
├── Makefile
├── main.tf
├── modules
│   ├── iam <-- the folder where the module cannot be resolved
│   │   └── main.tf
│   └── meta
│       └── outputs.tf
├── output.tf
├── readme.md
├── variables.tf
└── versions.tf

In the example we are downloading the module from S3, but it shouldn't matter what the source is. Notably, though, I haven't seen this happen with a local module reference (e.g. source = "../../path/to/module").

Steps to Reproduce

See the comment below https://github.com/VladRassokhin/intellij-hcl/issues/365#issuecomment-981972598

solarmosaic-kflorence commented 2 years ago

Here is a more thorough example. This is the project structure (this project exists in our monorepo repository at location: ./libraries/terraform/aws/bucket-object):

.
├── main.tf
├── modules
│   └── policies
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── outputs.tf
├── readme.md
├── variables.tf
└── versions.tf

Inside main.tf there is a module reference:

module "session" {
  source = "s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip"
}

Starting with no .terraform cache, I run terraform init in the root of the project...

➜  monorepo git:(feature/foo) cd libraries/terraform/aws/bucket-object
➜  bucket-object git:(feature/foo) terraform init
Initializing modules...
Downloading s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip for session...
- session in .terraform/modules/session

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v3.61.0...
- Installed hashicorp/aws v3.61.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The project .terraform cache folder now has the following structure:

.
├── modules
│   ├── modules.json
│   └── session
│       ├── main.tf
│       ├── modules
│       │   ├── meta
│       │   │   └── outputs.tf
│       │   └── policies
│       │       ├── main.tf
│       │       └── outputs.tf
│       ├── outputs.tf
│       └── versions.tf
└── providers
    └── registry.terraform.io
        └── hashicorp
            └── aws
                └── 3.61.0
                    └── darwin_amd64
                        └── terraform-provider-aws_v3.61.0_x5

The modules/modules.json file has the following content:

{"Modules":[{"Key":"","Source":"","Dir":"."},{"Key":"session","Source":"s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip","Dir":".terraform/modules/session"}]}

IntelliJ now correctly resolves this module from the cache, so the modules cache at the root of the project work fine. Now let's resolve the module references for the ./modules/policies sub-module, which contains the following in it's main.tf file:

module "session" {
  source = "s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip//modules/policies"
}

Again starting from having no .terraform cache, I navigate into the sub-module folder and run terraform init:

➜  bucket-object git:(feature/foo) cd modules/policies
➜  policies git:(feature/foo) terraform init
Initializing modules...
Downloading s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip for session...
- session in .terraform/modules/session/modules/policies

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The ./modules/policies/.terraform cache directory now has the following structure:

.
└── modules
    ├── modules.json
    └── session
        ├── main.tf
        ├── modules
        │   ├── meta
        │   │   └── outputs.tf
        │   └── policies
        │       ├── main.tf
        │       └── outputs.tf
        ├── outputs.tf
        └── versions.tf

The modules/modules.json file has the following content:

{"Modules":[{"Key":"","Source":"","Dir":"."},{"Key":"session","Source":"s3::https://s3.amazonaws.com/<REDACTED>/terraform/terraform-aws-session/terraform-aws-session_v0.2.2.zip//modules/policies","Dir":".terraform/modules/session/modules/policies"}]}

However, IntelliJ cannot resolve the module: Screen Shot 2021-11-29 at 2 04 10 PM

ambis commented 2 years ago

Exactly the same problem.

terraform plan works A-OK but PHPStorm doesn't find the module files

EDIT: This got fixed by just restarting the IDE.

Screenshot 2021-12-03 at 22 26 09

Xerkus commented 2 years ago

I believe it worked with multiple terraform modules in same project before and now it can't locate modules for unknown reason for all but one module.

mohamed-haidara-cko commented 2 years ago

Not sure if it's the same cause but had the same issue with Terraform 1.1.0. They changed how they store the Source in .terraform/modules/modules.json causing my issue. See https://github.com/hashicorp/terraform/pull/28854. Looks like the plugin reads that file.

Fixed it by making sure the Source field in .terraform/modules/modules.json matches exactly the source in the TF file. If doesn't match, modify it for all the modules in that file and restart the IDE (or close and open the file).

Example for an official module: change registry.terraform.io/terraform-aws-modules/vpc/aws to terraform-aws-modules/vpc/aws

I'm using PyCharm 2021.3 and version 0.7.14 of the plugin. I have also a monorepo with multiple Terraform projects!

jzburns commented 2 years ago

Same issue. This seems to have started recently:

Webstorm: Build #WS-213.5744.224, built on November 27, 2021 Plugin 0.7.14 OS: Arch Linux Error Cannot locate module locally: Unknown reason

module "gce-container-c1" {
  source = "terraform-google-modules/container-vm/google"
  version = "~> 2.0"
  container = {
    image = "eu.gcr.io/XXXX"
  }
  restart_policy = "Always"
}

Any workarounds?

Xerkus commented 2 years ago

Not sure if it's the same cause but had the same issue with Terraform 1.1.0. They changed how they store the Source in .terraform/modules/modules.json causing my issue. See hashicorp/terraform#28854. Looks like the plugin reads that file.

I can confirm, this solved the issue for me. Since the original reporter used 1.0.10, what we experienced here is a separate issue

danpoland commented 2 years ago

Alternativlity, you can prepend the source URI in your main.tf with registry.terraform.io/. Not including it is just a short hand way of indicating that the module is hosted on the Terraform registry. By prepending the registry in your source reference you don't have to touch the files in the cache.

module "vpc" {
  source  = "registry.terraform.io/terraform-aws-modules/vpc/aws"
  version = "3.11.0"
}
lonegunmanb commented 2 years ago

Thanks @danpoland , it's really helpful!

shuliakovsky commented 2 years ago

looks like plugin successfully can find a remote modules after terraform get with terraform version 1.0.11 and below, unfortunately can't find modules with latest terraform versions.

mohamed-haidara-cko commented 2 years ago

@shuliakovsky see my reponse above in case you missed it for a workaround: https://github.com/VladRassokhin/intellij-hcl/issues/365#issuecomment-996019841.

You will to need to update the .terraform/modules/modules.json for the plugin to work properly when using Terraform >= 1.1.0.

gchamon commented 2 years ago

We can use sed to quickly fix a repository with multiple workspaces:

sed -i --regexp-extended "s/registry.terraform.io\///g" **/.terraform/modules/modules.json && sed -i --regexp-extended "s/git::https:\/\/(.*)\.git/\1/" **/.terraform/modules/modules.json

I set an alias to this command so I can easily rerun it.

solarmosaic-kflorence commented 2 years ago

Please see https://github.com/VladRassokhin/intellij-hcl/issues/373 for issues related to the module paths introduced in Terraform 1.1.x -- the issue in this ticket only relates to sub-modules.

rashidnhm commented 2 years ago

Would there be a way to update the plugin to look for the new Source address alongside the exact match?

From glancing at hashicorp/terraform#28854 I see the following:

^By no means is this an exhaustive list, I compiled it from looking at some of the updated unit tests.

The fixes mentioned above do work for me, but it is a lot more verbose than I'd like. If the plugin can search modules.json for both an exact match OR an expected transformed match, wouldn't it handle the cases for both Terraform 1.1.x and below that as well?

userbradley commented 2 years ago

I've just recently had this issue, again despite checking that the values under source = and values under the file modules.json are the same

Resolution

  1. Try restarting Intelli-j, or
  2. See if there are any updates to the plugin, or
  3. Update terraform (Personally I use tfswitch, or
  4. Reboot ( ͡° ͜ʖ ͡°)

I'm not sure why this happened, but the above steps seem to solve it temporarily

fabio-filho commented 2 years ago

Not sure if it's the same cause but had the same issue with Terraform 1.1.0. They changed how they store the Source in .terraform/modules/modules.json causing my issue. See hashicorp/terraform#28854. Looks like the plugin reads that file.

Fixed it by making sure the Source field in .terraform/modules/modules.json matches exactly the source in the TF file. If doesn't match, modify it for all the modules in that file and restart the IDE (or close and open the file).

Example for an official module: change registry.terraform.io/terraform-aws-modules/vpc/aws to terraform-aws-modules/vpc/aws

I'm using PyCharm 2021.3 and version 0.7.14 of the plugin. I have also a monorepo with multiple Terraform projects!

I have same issue using:

This answer still works, thanks!

lozuwa commented 1 year ago

In my case I had to switch from terraform-aws-modules/vpc/aws to registry.terraform.io/terraform-aws-modules/vpc/aws.

psantus commented 1 year ago

In my case I had to switch from terraform-aws-modules/vpc/aws to registry.terraform.io/terraform-aws-modules/vpc/aws.

  • terraform 1.2.4
  • Intellij 2022.1.2

That tip was life-saving for me, thanks @lozuwa

mirzazeyrek commented 1 year ago

I can confirm the issue is still there and the solution mentioned above works.

https://plugins.jetbrains.com/plugin/7808-terraform-and-hcl v222.4345.14 PhpStorm 2022.2.3 Terraform v1.3.3 on linux_amd64