On Windows, Terraform can't resolve symlinks to volumes that don't have DOS-style drive letters

[apparentlymart]

Due to upstream Go issue golang/go#39786, any time Terraform uses filepath.EvalSymlinks it will fail on Windows systems where the given symlink refers to a UNC-style path to a volume, and where that volume doesn't have a DOS drive letter associated to it.

UNC-style raw paths are those starting with \\?\, which bypass the Win32 layer's attempts to apply DOS-style path mapping (via a mapping table of drive letters in the object store) and instead just pass the string directly to the kernel.

The Go standard library seems to assume that it's always possible to unpack such a path into an equivalent DOS-style path, but that isn't true: \\?\ paths can be used to refer to items that are not addressable by DOS-style paths, either due to syntax constraints (filenames containing characters that are reserved under DOS) or due to the target object not being included in the DOS devices mapping table at all.

On modern Windows, particularly in contexts involving virtualization, it's common to make a volume appear as if it is a subdirectory of another "drive" (in the DOS sense), rather than assigning it a drive letter directly, either to give that volume a more symbolic path name or to support scenarios where the system needs to interact with more than 26 distinct volumes and thus can't assign all of them a unique drive letter. Users achieve that by creating a symlink to a UNC-style raw volume path, such as the \?\Volume{1b3b1146-4076-11e1-84aa-806e6f6e6963}\ example shown in the upstream Go issue.

All of this comes together to mean that Terraform users can run into a number of strange errors when trying to use Terraform in directories that the user has accessed via a symlink to a UNC-style raw path, because the Go standard library fails with an error if Terraform calls filepath.EvalSymlinks (directly or indirectly) on such a path.

Terraform uses symlinks itself in a few spots, including:

• Avoiding making copies of plugin package directories in the global plugin cache directory, by symlinking from .terraform/... to there.
• Attempting to optimize against repeated downloads of the same remote module source address, by symlinking subsequent references to the directory created in response to the first reference.

Terraform also sometimes uses filepath.EvalSymlinks against symlinks it wasn't responsible for creating, which would presumably exhibit the same problems.

The Go team still seems to be working on understanding the root of this problem over in golang/go#39786, so for the moment this issue is here just to track that upstream blocker. We don't have enough context at our end to know the root cause here, and so we're monitoring the upstream issue to see where that leads before we take any actions on our end.

[apparentlymart]

Apparently, another situation where this can occur is when working with directories inside a OneDrive mount point, which I gather internally uses either the same or a similar mechanism as described here, where the result is a symlink that filepath.EvalSymlinks doesn't understand how to resolve.

[apparentlymart]

After some further research, it seems like there's some early consensus that what filepath.EvalSymlinks does -- particularly on Windows, but also on other platforms -- is not a great idea and that applications should typically be more selective in what sorts of normalization/expansion they do to paths.

I broadly agree with that position... it's confusing if e.g. you write a path down one way and an application returns an entirely different path that just happens to mean the same thing in the current execution context.

With that said, it's not clear yet what exactly we would change in Terraform if we decided to move forward in line with that consensus. We have various different reasons to use filepath.EvalSymlinks, some of which are in place to work around oddities of platforms other than Windows, so I think we'll need to review each call to filepath.EvalSymlinks on a case-by-case basis, understand what that usage was aiming to achieve, and evaluate whether it might make sense to remove that call or replace it with something else.

While I'm sure there will be some common themes, I don't expect that there's a general systematic answer where there'll just be a single drop-in replacement for all uses of filepath.EvalSymlinks.

There are various issues in the Go repository discussing different parts of this problem but I think golang/go#40180 is a good anchor point where those discussions aggregated into a proposal to add some additional context to the documentation of that function.

[elvis2]

What about adding a function in terraform that allows you to move files into the workspace in a tf init invocation?

For example, if we have a terraform library of modules, each module needs to have it's own provider block. Over time, ensuring all of these providers have the same content takes additional management time. Instead, I would like to have a single provider.tf in our library root that are symlink-ed.

Directory Structure:

library │   ├── modules │   ├── providers.tf

│   │   ├── application │   │   │   ├── alb │   │   │   │   ├── README.md │   │   │   │   ├── data.tf │   │   │   │   ├── main.tf │   │   │   │   ├── outputs.tf │   │   │   │   ├── providers.tf -> ../../../providers.tf │   │   │   │   ├── r53.tf │   │   │   │   └── vars.tf │   │   │   ├── asg │   │   │   │   ├── README.md │   │   │   │   ├── data.tf │   │   │   │   ├── iam.tf │   │   │   │   ├── main.tf │   │   │   │   ├── outs.tf │   │   │   │   ├── providers.tf -> ../../../providers.tf │   │   │   │   └── vars.tf

Since symlinks aren't a choice for team members who are on windows, maybe adding functions that can run during a terraform init to execute prepocesses? Normally you can run these types of preprocesses in a CICD environment, but if you aren't able to do this, and since the golang has issues with symlinks, either (a) copying the files into each directory or (b) executing a bash/python script to handle this. So the proposed file structure would be like:

library │   ├── modules │   ├── providers.tf

│   │   ├── application │   │   │   ├── alb │   │   │   │   ├── README.md │   │   │   │   ├── data.tf │   │   │   │   ├── main.tf │   │   │   │   ├── outputs.tf │   │   │   │   ├── r53.tf │   │   │   │   ├── providers.init.tf │   │   │   │   └── vars.tf │   │   │   ├── asg │   │   │   │   ├── README.md │   │   │   │   ├── data.tf │   │   │   │   ├── iam.tf │   │   │   │   ├── main.tf │   │   │   │   ├── outs.tf │   │   │   │   ├── providers.init.tf │   │   │   │   └── vars.tf

Where *.init.tf would get special treatment in the golang and process the instructions within the .init.tf. Example:

providers.init.tf:

resource terraform_init_copy "copy_providers" { source: '../../../providers.tf' target: '.' }

[apparentlymart]

Hi @elvis2,

There's lots of precedent in the community of using preprocessing steps in conjunction with Terraform to achieve various sorts of templating and reuse, but existing patterns typically involve having the external tool run Terraform rather than having Terraform run the external tool. If the focus is only on work related to terraform init then it's possible to write a wrapper that only wraps terraform init, and then to run the normal Terraform commands directly for the rest of the workflow.

I don't expect that we would add features to run arbitrary external software during terraform init, because that would significantly increase the possible attack surface of installing a third-party module. Today terraform init is careful not to immediately execute anything it has installed in order to give users an opportunity to review what was installed before running it; other commands like terraform apply are Terraform's first opportunity to execute code from either provider packages or module packages.

[robpomeroy]

I've encountered a related issue when running Terraform from Windows on a WSL file path. Looks like terraform init invokes CMD.EXE, even when run from PowerShell. And CMD.EXE can't handle UNC paths so ends up in the wrong directory.

PS Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\AlmaLinux9\mnt\wsl\repos\some-tf-repo> terraform version
Terraform v1.8.5
on windows_386
PS Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\AlmaLinux9\mnt\wsl\repos\some-tf-repo> terraform init    

Initializing the backend...
'\\wsl.localhost\AlmaLinux9\mnt\wsl\repos\some-tf-repo'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
╷
│ Error: Error locking state: [{%!s(tfdiags.Severity=69) Error acquiring the state lock Error message: 2 errors occurred:
│       * Incorrect function.
│       * open .terraform\.terraform.tfstate.lock.info: The system cannot find the file specified.
│
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended. }]
│
│
╵

[apparentlymart]

Hi @robpomeroy,

Thanks for reporting that. It seems like something quite different than what this issue was about, so it would be helpful if you would open a new issue and complete the issue template so that we can try to reproduce this and understand exactly which part of the init process is running the Windows command interpreter and why it is doing that. Thanks!

[robpomeroy]

Sure thing - my bad!

[apparentlymart]

The upstream Go issue https://github.com/golang/go/issues/39786 seems to have been fixed for Go 1.23 by no longer treating reparse points as if they are symlinks. That then causes the EvalSymlinks function not to try to resolve those, thereby avoiding the error this issue was about.

Go 1.23 is not yet release and so Terraform cannot yet adopt it. Unfortunately for this round the Terraform release cycle completed before the Go one and so Terraform v1.9 will remain on Go 1.22 and our earliest opportunity to upgrade to Go 1.23 will be for the Terraform v1.10 release. Once the main branch is updated to specify a Go 1.23 release I think we should be able to close this issue.