[LFX] Enhancement: KCL Package Management Dependencies Sparse Checkout

[zong-zhe]

Enhancement

kpm is a package management tool for KCL. When the scale of KCL project becomes larger and larger, and the external packages that KCL project relies on become more and more, kpm will become slow due to the need to download a large number of third-party dependencies. kpm needs to support Sparse-Checkout, which means downloading specific dependencies as needed rather than all of them, to improve the performance of the kpm.

kpm currently supports managing third-party library dependencies through Git repositories. It requires that a kcl.mod file must be present at the root directory of the Git repository. kpm treats the entire Git repository as a single KCL package. This approach presents performance problem when dealing with monorepos, which may contain many KCL packages. The current KCL program might only depend on one of these packages, and KPM would only need to download that specific package, not the entire repository. Therefore, when adding Git repositories as third-party dependencies in kpm, there is a need to support adding a subdirectory of a Git repository as the third-party dependency.

• Pretest 1: A design document, first of all, we need to design how users use this feature, we need to design this feature in the kpm command line interface, that is, how users through the kpm command line and kcl.mod, add a git repository subdirectory as a third-party dependency.

• Pretest 2: A kpm development work, to help you more familiar with the kpm project, support for dependency renaming, kpm does not currently support dependency renaming, the name of the dependency must be the same as the name field in the kcl.mod file. This can be confusing for users, so we need to support renaming that allows users to rename three-party dependencies. https://github.com/kcl-lang/kpm/issues/288

• Pretest 3: Refactoring the downloading for KCL dependencies from the git repo via GitDownloader. https://github.com/kcl-lang/kpm/issues/344

[d4v1d03]

What's the pre-test to this issue @zong-zhe , @Peefy ?

[zong-zhe]

Hi @d4v1d03 😄 Welcome ! I update the pretest for this issue.

[officialasishkumar]

├── my_package
│   ├── a.xyz
│   ├── b.xyz
│   ├── kcl.mod
│   ├── kcl.mod.lock
│   └── main.k
├── my_package_1
│   ├── kcl.mod
│   ├── kcl.mod.lock
│   └── main.k

If this is the entire working directory of my monorepo for KCL, will sparse checkout isolate a directory, such as my_package_1, when running KPM commands from the root?

Additionally, please let me know if there are any specific functionalities required.

cc: @zong-zhe @Peefy

[officialasishkumar]

Hey,

How do you want me to send the design doc? Through email or a pull request?

cc: @zong-zhe @Peefy

[zong-zhe]

Hey,

How do you want me to send the design doc? Through email or a pull request?

cc: @zong-zhe @Peefy

Hi 😀，PR for a markdown is OK.

[zong-zhe]

Hi @officialasishkumar @d4v1d03 😄

I have added a new Pretest 3, which is responsible for refactoring the code of kpm downloading third-party dependencies from git repositories. This Pretest 3 will not change any feature, and this Pretest will help you to have a deeper understanding of the current status of kpm. Including code implementation, unit testing, integration test, etc., complete refactoring, and make it available at https://github.com/kcl-lang/cli