No semantic tokens returned for a function call

[rcjsuen]

This HCL file seems pretty straightforward to me but I don't have any semantic tokens for functions. Is this intentional?

group {
  name = lower("")
}

[{hcl-blockType [] example.tf:1,1-6} {hcl-attrName [] example.tf:2,3-7}]

package main

import (
    "context"
    "fmt"

    "github.com/hashicorp/hcl-lang/decoder"
    "github.com/hashicorp/hcl-lang/lang"
    "github.com/hashicorp/hcl-lang/schema"
    "github.com/hashicorp/hcl/v2"
    "github.com/hashicorp/hcl/v2/hclsyntax"
    "github.com/zclconf/go-cty/cty"
)

type PathReader struct {
    Files map[string]*hcl.File
}

func (pr *PathReader) PathContext(path lang.Path) (*decoder.PathContext, error) {
    schema := &schema.BodySchema{
        Blocks: map[string]*schema.BlockSchema{
            "group": {
                Labels: []*schema.LabelSchema{
                    {
                        Name: "groupName",
                    },
                },
                Body: &schema.BodySchema{
                    Attributes: map[string]*schema.AttributeSchema{
                        "name": {
                            IsOptional: true,
                            Constraint: schema.AnyExpression{OfType: cty.String},
                        },
                    },
                },
            },
        },
    }

    return &decoder.PathContext{Schema: schema, Files: pr.Files}, nil
}

func (pr *PathReader) Paths(ctx context.Context) []lang.Path {
    return []lang.Path{}
}

func main() {
    f, pDiags := hclsyntax.ParseConfig([]byte("group {\n  name = lower(\"\")\n}"), "example.tf", hcl.InitialPos)
    if len(pDiags) > 0 {
        panic(pDiags)
    }

    decoder := decoder.NewDecoder(&PathReader{Files: map[string]*hcl.File{"example.tf": f}})
    pathDecoder, err := decoder.Path(lang.Path{Path: ".", LanguageID: "terraform"})
    if err != nil {
        panic(err)
    }

    tokens, err := pathDecoder.SemanticTokensInFile(context.Background(), "example.tf")
    if err != nil {
        panic(err)
    }
    fmt.Println(tokens)
}

[dbanck]

Hi @rcjsuen,

I believe we only add semantic tokens for known function calls. The PathContext maintains a list of all available functions and if you add lower to that list, you should see the expected semantic token.

return &decoder.PathContext{
    Schema: bodySchema,
    Files:  pr.Files,
    Functions: map[string]schema.FunctionSignature{
        "lower": {
            Params: []function.Parameter{
                {
                    Name: "str",
                    Type: cty.String,
                },
            },
            ReturnType:  cty.String,
            Description: "`lower` converts all cased letters in the given string to lowercase.",
        },
    },
}, nil

Result

[{hcl-blockType [] example.tf:1,1-6} {hcl-attrName [] example.tf:2,3-7} {hcl-functionName [] example.tf:2,10-15}]

I hope this helps!

[rcjsuen]

I believe we only add semantic tokens for known function calls.

That is very surprising. The parser knows what a function looks like so it is not clear to me why the name of the functions must be registered first? 🤔

[radeksimko]

Considering one of the main roles of this library is to build language servers for HCL-based languages it also means it provides functionality relevant to that HCL-based language, rather than HCL.

The parser knows what a function looks like so it is not clear to me why the name of the functions must be registered first?

The problem "what looks like a function" is valid but it is solved by simpler syntax grammars, such as TextMate - see https://github.com/hashicorp/syntax Most IDEs rely on TextMate or comparable (simpler) grammars (as opposed to parsers) to do the same job.

As you rightly said the parser knows what a function looks like but on its own (without schema) doesn't know if it is in fact valid function, or even function call (as opposed to type constraint). HCL has an (opt in) concept of type constraint that make use of function calls - see https://developer.hashicorp.com/terraform/language/expressions/type-constraints#structural-types

variable "example" {
  type = map(string)
}

and it is exactly the role of semantic highlighting to be able to tell the difference between the two.

---

All that said, I guess there could be some opt-in behaviour where the tokens returned don't take valid function names into account. This would only be relevant in a context where there is no better highlighting mechanism mentioned above and semantic highlighting is the only way to highlight AND the highlighting is being done in the context of plain HCL (e.g. https://marketplace.visualstudio.com/items?itemName=HashiCorp.HCL) rather than the HCL-based language like Terraform.

@rcjsuen Have you come across such a context?

[rcjsuen]

@rcjsuen Have you come across such a context?

Not sure if this answers your question but ultimately I want the language server to colour everything so that the plug-in into the language client is as simplest as possible.