Syntax highlighting is not working reliably

[Rabattkarte]

Versions

This bug is reproducible in:

• [x] the latest version of the extension (below)
• [x] the latest version of the language server (below)

Extension

v2.19.0

Language Server

terraform-ls 0.25.2

VS Code

Version: 1.64.2
Commit: f80445acd5a3dadef24aa209168452a3d97cc326
Date: 2022-02-09T22:00:58.347Z
Electron: 13.5.2
Chromium: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 20.6.0

Operating System

macOS Big Sur 11.6.2 (20G314)

Terraform Version

$ terraform version
Terraform v1.1.5
on darwin_amd64

Steps To Reproduce

Steps to reproduce the behavior:

1. Create a main.tf with the content of the screenshots, the syntax highlight does not work.
2. The default language will be "Terraform"
3. Hit Ctrl+Shift+P and "Change Language Mode -> terraform-vars"
4. The syntax highlight works

This is weird and feels not intended.

Include any relevant Terraform configuration or project structure:

resource "azurerm_resource_group" "main" {
  name     = local.resource_group_name
  location = var.location

  tags = local.all_tags
}

Please, see screenshots attached.

Weirdly working terraform-vars:

Not working Terraform:

Expected Behavior

Syntax highlighting should just work. Why are there two languages anyway?

Actual Behavior

Syntax highlighting does not work. :(

Additional context

Extension bisect/only enabling Hashicorp Terraform extension did not fix the problem.

[dbanck]

Thanks for reporting this issue, @Rabattkarte!

You're right; this is not intended behavior. Sadly I'm not able to reproduce the reported issue.

Can you do me a favor and post a screenshot with the scope inspector open and the cursor on a var or local keyword of your non-highlighted main.tf? This will show which scopes are applied to the selected keyword and which are missing.

[dbanck]

Expected Behavior

Syntax highlighting should just work. Why are there two languages anyway?

We are using two language ids:

• terraform - standard *.tf config files
• terraform-vars - variable files (*.tfvars)

We need this because both kinds of files have different meanings in a Terraform project. The Terraform language server running in the background needs to know which type of file it's parsing, so things like goto-variable can work. Also, .tfvars only allow a subset of the syntax allowed in a .tf file; for example, blocks aren't allowed.

---

The configured color theme might also affect which keywords are highlighted. Do you mind sharing which theme you're using?

[Rabattkarte]

Hi @dbanck, thanks for your swift reply and explaination!

The configured color theme might also affect which keywords are highlighted. Do you mind sharing which theme you're using?

I'm using Dracula Official. But the same syntax problems occur with every other theme as well. Even Default (dark & light).

---

Do you know of any other meaningful ways I could provide debug logs? I googled already how to log extension behavior and/or what extensions are invoked when I open an editor. Sadly I did not find an answer.

[Rabattkarte]

Can you do me a favor and post a screenshot with the scope inspector open and the cursor on a var or local keyword of your non-highlighted main.tf? This will show which scopes are applied to the selected keyword and which are missing.

@dbanck Sorry, I totally missed your response. Here's my result. What's strange is that it is detected as language terraform-vars, which is wrong if I got your explanation right. Yes, I set the language for the main.tf to be Terraform, not terraform-vars.

[dbanck]

Thank you for providing the screenshot. This is very helpful and somewhat confusing 😉

Do you have something in your settings for file associations? Like this:

"files.associations": {
  "*.tf": "terraform-vars"
}

This would explain why a .tf file has the wrong language id.

[jpogran]

Looking at your screenshot shows there's two interactions at play here competing to provide syntax highlighting. The textmate scopes section shows that the Terraform grammar is detected and being applied. This is correct, and what we expect to see

The second section semantic token type comes from the language server and is meant to provide more specific tokens to determine which part is highlighted. Notice the perl, ts, js, and tsx at the end of the scopes. These indicate that these semantic tokens are coming from some other extension that is operating in addition to ours. Semantic tokens always 'win' over textmate scopes, so whatever other extension is providing this will always override what we are providing.

Something is either mis-configured or there is an extension claiming ownership of Terraform files that is not us. Please list out the extensions installed (you can run code --list-extensions on the commandline) and any settings in your editor that control file associations like @dbanck showed.

[Rabattkarte]

@dbanck I have checked all configs:

• ~/Library/Application Support/Code/User/settings.json
• my.code-workspace (the file I'm using for this workspace) and found no files.associations block.

But, I restarted and reloaded vscode several times and now it is displaying terraform as the language. The semantic tokens are still displayed, though.

---

@jpogran thank you very much for your explanation. TIL! Here's the list of all installed extensions:

$ code --list-extensions
bierner.markdown-emoji
bungcip.better-toml
dakara.transformer
DavidAnson.vscode-markdownlint
dracula-theme.theme-dracula
eamodio.gitlens
EditorConfig.EditorConfig
esbenp.prettier-vscode
foxundermoon.shell-format
Gruntfuggly.todo-tree
hashicorp.terraform
hilleer.yaml-plus-json
mechatroner.rainbow-csv
mhutchie.git-graph
ms-azure-devops.azure-pipelines
ms-azuretools.vscode-docker
ms-dotnettools.vscode-dotnet-runtime
ms-kubernetes-tools.vscode-aks-tools
ms-kubernetes-tools.vscode-kubernetes-tools
ms-vscode.azure-account
ms-vscode.powershell
msazurermtools.azurerm-vscode-tools
oderwat.indent-rainbow
PascalReitermann93.vscode-yaml-sort
redhat.vscode-yaml
richie5um2.vscode-sort-json
streetsidesoftware.code-spell-checker
streetsidesoftware.code-spell-checker-german
timonwong.shellcheck
tuxtina.json2yaml
vscode-icons-team.vscode-icons
yzhang.markdown-all-in-one

The following three extensions are disabled, though:

eamodio.gitlens
mechatroner.rainbow-csv
mhutchie.git-graph

I went through all extensions and had a look at what language features they are contributing, but I was unable to identify the offending extension.

What's interesting is if I disable all extensions except for HashiCorp Terraform, and remove my settings.json and the workspace file. I am still facing that issue. Please have a look at the following screenshot:

Do you have any more hints or suggestions? I am really puzzled.

[Rabattkarte]

One thing I just noticed. This bug apparently only happens in data and resource blocks. locals blocks render just fine.

Here is my terraform/versions block, if this helps?

terraform {
  required_providers {
    azurerm = {
      source                = "hashicorp/azurerm"
      version               = "~> 2.46"
      configuration_aliases = [azurerm.shared]
    }
  }

  required_version = ">= 0.12"
}

---

Now, I checked by adding the AWS provider and creating a resource. And guess what, highlighting seems to be working...

Maybe that's a bug in the hashicorp/azurerm provider. I saw something about a downloaded schema in some log file. Might this be a hint?

[dbanck]

We looked into this today and were somewhat able to reproduce the behavior.

Some background: As James said, we rely on TextMate grammars and semantic highlighting for syntax highlighting. TextMate grammars work on a single file as input and break it up based on lexical rules expressed in regular expressions. Highlighting based on semantic tokens is considered an addition to the TextMate-based syntax highlighting and goes on top of syntax highlighting. As language servers can take a while to load and analyze a project, semantic token highlighting may appear after a short delay.

The current state of the static grammar highlights the var keyword different from the .variable_name.

When the language server is done processing all files, we have more information about the available variables and can adjust highlighting accordingly.

See how the name line is highlighted differently from the location one? That's because the LS can't find a variable called non_existing. This might also be the case for var.something in your AWS example.

---

I think there is no immediate fix for this with all that in mind. But there is a lot of room for improvement in the static grammar and the semantic token side. Highlighting should be more consistent and intuitive. So we will create separate issues for each enhancement.

Thanks again for bringing this to our attention and providing detailed feedback! 🙏

[Rabattkarte]

Oh well, with your explanation I just realized that I had completely wrong (other?) assumptions about what the extension provides. I thought the keyword var. will be highlighted to aid with visual reviews. But now I understood that e.g. var.existing_var will be displayed inline (without any highlighting) since this is a known reference. This becomes even clearer when I hover over that variable.

---

I really learned a lot about debugging vscode extensions as well as how they work internally. I guess there is nothing to "fix" in this issue. I see exactly the same improvement that you mentioned:

Highlighting should be more consistent and intuitive. So we will create separate issues for each enhancement.

I'm closing this issue now since it is none. Feel free to re-open it if needed.

Thanks again! 🥇

[dbanck]

FYI, I've created two issues tracking the described problems:

• hashicorp/syntax#24
• https://github.com/hashicorp/terraform-ls/issues/805

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.