search

kcl-lang / kcl

KCL Programming Language (CNCF Sandbox Project). https://kcl-lang.io
https://kcl-lang.io
Apache License 2.0
1.69k stars 119 forks source link

[Track] KCL IDE/LSP workspace design #1510

Open He1pa opened 4 months ago

He1pa commented 4 months ago

KCL IDE/LSP workspace

Main Problem

KCL's CLI specifies the compilation entry by inputting a file, files list or a configuration file by argument -Y But IDE and LSP dont know what to compile and analyze

Cases

Now we are facing several cases

1. Without configuration file

-- pkg
  -- a.k
  -- b.k

2. kcl.yaml

This pattern appears in the internal, closed-source konfig repository

-- kcl.mod (kcl v0.4.3, it is not the `kcl.mod` in current kpm)
-- base
  -- pkg1
    -- a.k
  -- pkg2
-- prog1
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- project.yaml
-- prog2
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
   -- project.yaml

kcl.yaml

kcl_cli_configs:
  file:
    - base/pkg1.a.k
    - ../base/base.k
    - main.k

Compile with kcl run -Y kcl.yaml

3.kcl.mod

-- pkg
  -- a.k
-- main.k
-- kcl.mod

kcl.mod

[package]
name = "xxx"
version = "0.1.0"

[dependencies]
xx = { .. }
[profile]
entries = ["main.k"]

Current processing logic

Problem

For case 2 and 3, IDE/LSP will only get the correct project structure when processing main.k. But when processing pkg/a.k, it will be treated as case 1, and pkg will be used as the entry for compile. At this time, the lack of information about kcl.mod and main.k will lead to incomplete semantic information, and features such as find_ref and rename will be unavailable.

-- pkg
  -- a.k   -> GlobalState 1
-- main.k  
-- kcl.mod -> GlobalState 2

In addition, when we compile with kcl.mod as the entry, and then open pkg/a.k.Aactually, it has been compiled and analyzed. But according to the current logic, it will be recompiled, which will also cause performance loss in IDE.

In a workspace, there are multiple entries at the same time. In order to save the status of multiple entries at the same time, LSP will also have performance loss. https://github.com/kcl-lang/kcl/issues/1237

~## Solution~ ~LSP set two modes: SingleEntry and MultiEntry~

~Try to find kcl.mod in the IDE workspace~

~The logic is consistent with previous to be compatible with the konfig repository. New KCL projects are recommended to use kcl.mod~

~### LSP Changes~

~Previously, opened/changed a file -> found compile entry -> compile -> update GlobalState~

~After the change, Start LSP -> found compile entry -> compile -> update GlobalState -> update the GlobalState cache according to the changed file~

~### Impact on lsp cache/incremental compilation~

~In the previous design, no matter what the case, the changed files are in the main pkg, and the incremental compilation of resolver/advanced_resolver is based on the premise that the changed files are in the main pkg.~

~After changing the entry point, it is necessary to reconsider the processing of editing files that are not in the main pkg.~

~For the Parse stage:~ ~The cache is a map of filename -> AST. There is no pkg, so no need to update~

~For the Resolver stage:~ ~Previously, the resolver needed to clear the cache of main_pkg and the affected pkgs. After the change, it is necessary to first query which pkg the file belongs to, analyze the affected pkgs, and clear the cache of these pkgs.~

~For Namer and Advanced Resolver:~ ~Update the pkg that was updated in the resolver stage, no changes are required~

KCL/KPM run

KCL/KPM run also has similar logic when executing, and needs to find the compilation entry and config from the input. The relevant implementation is in https://github.com/kcl-lang/kpm/blob/2c16041e969cd2f7cdaf3af4e4d0cf7d96c67003/pkg/client/run.go#L409

// getPkgSource returns the package source.
// Compiling multiple packages at the same time will cause an error.
func (o *RunOptions) getPkgSource() (*downloader.Source, error) {
    workDir := o.WorkDir

    var pkgSource *downloader.Source
    if len(o.Sources) == 0 {
        workDir, err := filepath.Abs(workDir)
        if err != nil {
            return nil, err
        }
        // If no sources set by options, return a localSource to facilitate searching for
        // compilation entries from configurations such as kcl.yaml and kcl.mod files.
        return &downloader.Source{
            Local: &downloader.Local{
                Path: workDir,
            },
        }, nil
    } else {
        var rootPath string
        var err error
        for _, source := range o.Sources {
            if pkgSource == nil {
                pkgSource = source
                rootPath, err = source.FindRootPath()
                if err != nil {
                    return nil, err
                }
            } else {
                rootSourceStr, err := pkgSource.ToString()
                if err != nil {
                    return nil, err
                }

                sourceStr, err := source.ToString()
                if err != nil {
                    return nil, err
                }

                if pkgSource.IsPackaged() || source.IsPackaged() {
                    return nil, reporter.NewErrorEvent(
                        reporter.CompileFailed,
                        fmt.Errorf("cannot compile multiple packages %s at the same time", []string{rootSourceStr, sourceStr}),
                        "only allows one package to be compiled at a time",
                    )
                }

                if !pkgSource.IsPackaged() && !source.IsPackaged() {
                    tmpRootPath, err := source.FindRootPath()
                    if err != nil {
                        return nil, err
                    }
                    if tmpRootPath != rootPath {
                        return nil, reporter.NewErrorEvent(
                            reporter.CompileFailed,
                            fmt.Errorf("cannot compile multiple packages %s at the same time", []string{tmpRootPath, rootPath}),
                            "only allows one package to be compiled at a time",
                        )
                    }
                }
            }
        }

        // A local k file or folder
        if pkgSource.IsLocalKPath() || pkgSource.IsDir() {
            rootPath, err = pkgSource.FindRootPath()
            if err != nil {
                return nil, err
            }

            pkgSource, err = downloader.NewSourceFromStr(rootPath)
            if err != nil {
                return nil, err
            }
        }
    }

    if pkgSource == nil {
        return nil, errors.New("no source provided")
    }

    return pkgSource, nil
}

and

func (local *Local) FindRootPath() (string, error) {
    if local == nil {
        return "", fmt.Errorf("local source is nil")
    }

    // if local.Path is a directory, judge if it has kcl.mod file
    if utils.DirExists(filepath.Join(local.Path, constants.KCL_MOD)) {
        abspath, err := filepath.Abs(local.Path)
        if err != nil {
            return "", err
        }
        return abspath, nil
    }

    // if local.Path is a *.k file, find the kcl.mod file in the same directory and in the parent directory

    dir := filepath.Dir(local.Path)
    for {
        kclModPath := filepath.Join(dir, constants.KCL_MOD)
        if utils.DirExists(kclModPath) {
            abspath, err := filepath.Abs(kclModPath)
            if err != nil {
                return "", err
            }
            return filepath.Dir(abspath), nil
        }

        parentDir := filepath.Dir(dir)
        if parentDir == dir {
            break
        }
        dir = parentDir
    }

    // If no kcl.mod file is found, return the directory of the original file
    var abspath string
    var err error
    if local.IsLocalKPath() {
        abspath, err = filepath.Abs(filepath.Dir(local.Path))
        if err != nil {
            return "", err
        }
    } else {
        abspath, err = filepath.Abs(local.Path)
        if err != nil {
            return "", err
        }
    }

    return abspath, nil
}

There are two kinds of cases related to IDE

  1. workDir and IDE workspace. If there is no input, like 
    # Run the current package
kcl run
    if len(o.Sources) == 0 {
        workDir, err := filepath.Abs(workDir)
        if err != nil {
            return nil, err
        }
        // If no sources set by options, return a localSource to facilitate searching for
        // compilation entries from configurations such as kcl.yaml and kcl.mod files.
        return &downloader.Source{
            Local: &downloader.Local{
                Path: workDir,
            },
        }, nil
    }

This is similar to opening an IDE WorkSpace (but without opening any files), using the current directory(workspace in IDE) as the entry

  1. a signle input & edit a kcl file in IDE
for _, source := range o.Sources {
    if pkgSource == nil {
        pkgSource = source
        rootPath, err = source.FindRootPath()
        if err != nil {
            return nil, err
        }
    } else {}

Ref

Rust Analyzer

First, we need to figure out what to analyze. To do this, we run cargo metadata to learn about Cargo packages for current workspace and dependencies, and we run rustc --print sysroot and scan the "sysroot" (the directory containing the current Rust toolchain's files) to learn about crates like std. This happens in the [GlobalState::fetch_workspaces] method. We load this configuration at the start of the server in [GlobalState::new], but it's also triggered by workspace change events and requests to reload the workspace from the client.

  • Cargo.toml and Cargo.lock are stored in the root of your package (package root).
  • Source code goes in the src directory.
  • The default library file is src/lib.rs.
  • The default executable file is src/main.rs.
    • Other executables can be placed in src/bin/.
  • Benchmarks go in the benches directory.
  • Examples go in the examples directory.
  • Integration tests go in the tests directory.

Go:

The VS Code Go extension supports both GOPATH and Go modules modes.

Go modules are used to manage dependencies in recent versions of Go. Modules replace the GOPATH-based approach to specifying which source files are used in a given build, and they are the default build mode in go1.16+. We highly recommend Go development in module mode. If you are working on existing projects, please consider migrating to modules.

Unlike the traditional GOPATH mode, module mode does not require the workspace to be located under GOPATH nor to use a specific structure. A module is defined by a directory tree of Go source files with a go.mod file in the tree's root directory.

Your project may involve one or more modules. If you are working with multiple modules or uncommon project layouts, you will need to configure your workspace by using Workspace Folders. See the Supported workspace layouts documentation for more information.

He1pa commented 3 months ago

ref: https://dpe.org/how-your-monorepo-breaks-the-ide/

In Intellij IDEA we have long-standing assumption that one code file belongs to exactly one module and this module is compiled with a specific dependencies and has a specific compiler version and scpecific libiaries SO you can only really edit it in one context at once even though you would want to do something in multiple context

He1pa commented 3 months ago

https://github.com/golang/tools/blob/master/gopls/doc/workspace.md https://go.dev/blog/gopls-scalability

He1pa commented 3 months ago

Design

kcl.work

Reference from gopls's go work and rust's workspace in cargo.toml, use an explicit configuration file kcl.work to define the LSP workspace.

The kcl.work file is line oriented. Each line holds a single directive, made up of a keyword(there is only one keyword workspace) followed by arguments. For example:

workspace ./a
workspace ./b
workspace ./c/d

For User

In single workspace

Most kcl projects may not need kcl.work. They only work in one workspace and manage the directory structure through kpm.

kcl mod init
- kcl.mod
- kcl.mod.lock
- main.k

LSP workspace is equal to kcl run in current path

In addition to main.k, users can also define the compilation entry of the current pkg through the etries field in kcl.mod

In mult-workspace(Konfig)

It is almost impossible to maintain the semantic information of hundreds or thousands of workspaces in memory at the same time. Also, in the Konfig repository, users may not care about the code in other projects. Therefore, we need to explicitly use kcl.work to define the active workspace.

In the following project structure:

-- kcl.mod (kcl v0.4.3, it is not the `kcl.mod` in current kpm)
-- base
  -- pkg1
    -- a.k
  -- pkg2
-- prog1
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- project.yaml
-- prog2
  -- base
    -- base.k
  -- prod
    -- kcl.yaml
    -- main.k
    -- stack.yaml
  -- test
    -- kcl.yaml
    -- main.k
    -- stack.yaml
   -- project.yaml

Assuming that user A maintains prog1, he/she can create a kcl.work in the root directory and add the following code to kcl.work

workspace ./prog1/prod
workspace ./prog1/test

Functions like rename and find_ref will only work in stack prod and test.

It is generally inadvisable to commit go.work files into version control systems.

Compatibility Solutions

We recommend to set kcl.work or kcl.mod in the root directory to set up the workspace. But if not, use the directory of edit file. Search for compilation configuration and compilation entry in the current directory according to the following priority

For dev

Change on server