argocd kcl plugin rendering is slow and file.modpath is cached by different kcl modules

[riven-blade]

General Question

argo logs time="2024-06-19T05:40:54Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2024-06-19T05:40:41Z" grpc.time_ms=13448.232 span.kind=server system=grpc time="2024-06-19T05:41:36Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2024-06-19T05:41:23Z" grpc.time_ms=13530.473 span.kind=server system=grpc time="2024-06-19T05:41:50Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GetParametersAnnouncement grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2024-06-19T05:41:36Z" grpc.time_ms=13499.884 span.kind=server system=grpc

Plug-in configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: kcl-plugin-config
  namespace: argocd
data:
  # Sometimes, the ArgoCD runs the kcl run command twice simultaneously,
  # leading to a race condition in the usage of files inside the
  # KCL_CACHE_PATH and KCL_PKG_PATH directories.
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: kcl
    spec:
      version: v1.0
      discover:
        fileName: "./kcl.yaml"
      generate:
        command: [/home/argocd/generate.sh]

spec:
  template:
    spec:
      containers:
      - name: kcl-plugin
        command: [/tini]
        args:
          - /var/run/argocd/argocd-cmp-server
#          - --
#          - --loglevel=debug
        image: kcllang/kcllang-cmp-plugin
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          # Remove this volumeMount if you've chosen to bake the config file into the sidecar image.
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: plugin.yaml
            name: kcl-plugin-config
          # Starting with v2.4, do NOT mount the same tmp volume as the repo-server container. The filesystem separation helps
          # mitigate path traversal attacks.
          - mountPath: /tmp
            name: cmp-tmp
      volumes:
      - configMap:
          name: kcl-plugin-config
        name: kcl-plugin-config
      - emptyDir: {}
        name: cmp-tmp

images: kcllang/kcllang-cmp-plugin

shell

#!/bin/bash

# Set up temporary directories and file
export KCL_CACHE_PATH=$(mktemp -d /tmp/kcl_cache.XXXXXXXXXX)
export KCL_PKG_PATH=$(mktemp -d /tmp/kcl_pkg.XXXXXXXXXX)
tempfile=$(mktemp)

# Prepare KCL parameters from environment variables
KCL_PARAMS=""
for var in $(env); do
  if [[ "$var" == ARGOCD_ENV_param_* ]]; then
    var_name=${var%%=*}        # Extract the name of the variable up to the first '='
    var_value=${var#*=}        # Extract the value of the variable after the first '='
    clean_name=${var_name#ARGOCD_ENV_param_}
    KCL_PARAMS+="-D ${clean_name}=${var_value} "
  fi
done

# Trim the trailing space from KCL_PARAMS if necessary
KCL_PARAMS="${KCL_PARAMS% }"

# Run the KCL command with parameters and handle errors
if kcl run $KCL_PARAMS -q -o "$tempfile" 2>/dev/null; then
  cat "$tempfile"
  rm "$tempfile"
else
  error=$?
  echo "Error running kcl with code $error"
  exit $error
fi

[Peefy]

Hello @riven-blade I've updated the package cache path to /tmp in this PR https://github.com/kcl-lang/kcl-lang.io/pull/390 you can refer it to update the plugin config.

Thanks for the feedback.

[riven-blade]

It doesn't seem to work, it's still very slow.

[Peefy]

@riven-blade Your configuration seems to rely on external packages, but it doesn't seem to have placed the cache in the corresponding location. Have you changed the configuration of argocd kcl plugin and restarted argocd?

[riven-blade]

Yes, I changed the cache and pkg paths according to the issuer, and restarted argocd

[Peefy]

Yes, I changed the cache and pkg paths according to the issuer, and restarted argocd

Can you set this environment variable locally, run this configuration repeatedly locally, and tell me the time cost and show the ls /tmp output? Besides, what's the kcl version in the kcllang/kcllang-cmp-plugin image?

[riven-blade]

I have some ideas. I distinguish the stored files and pkg according to the angocd application. Each application is assigned a stored file, so that it should be able to be installed.

[Peefy]

export KCL_CACHE_PATH="/tmp/kcl_cache.${ARGOCD_APP_NAME}.${ARGOCD_APP_NAMESPACE}"
export KCL_PKG_PATH="/tmp/kcl_pkg.${ARGOCD_APP_NAME}.${ARGOCD_APP_NAMESPACE}"

create_dir_if_not_exists() {
  local dir_path=$1
  if [ ! -d "$dir_path" ]; then
    mkdir -p "$dir_path"
  fi
}

create_dir_if_not_exists "$KCL_CACHE_PATH"
create_dir_if_not_exists "$KCL_PKG_PATH"

[riven-blade]

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: /home/argocd/generate.sh failed exit status 1: EvaluationError failed to access the file '/tmp/_cmp_server/f48cbbfa-b6fb-445a-b94a-332cec8bdb83/kcl-tiangong/./datastore/staging/application-config.yaml': No such file or directory (os error 2)

The yaml file is referenced in the code, and the cache server has an absolute path.

[riven-blade]

base: tiangong.App {
    configMap = {"application-config.yaml": "./datastore/staging/application-config.yaml"}
    secret = {"application-secret.yaml": "./datastore/staging/application-secret.yaml"}
}

[Peefy]

Hello, you can use the file.modpath() function to get the path. For example

import file

base: tiangong.App {
    configMap = {"application-config.yaml": file.modpath() + "datastore/staging/application-config.yaml"}
    secret = {"application-secret.yaml":  file.modpath() + "datastore/staging/application-secret.yaml"}
}

[Peefy]

@riven-blade KCL caches external dependency package code and compilation cache, both of which are recorded in relative paths. If you are a custom YAML configuration file, please ensure that it exists within kcl. mod and use a path relative to kcl. mod in the code to read it. For example.

import file

base: tiangong.App {
    configMap = {"application-config.yaml": file.modpath() + "datastore/staging/application-config.yaml"}
    secret = {"application-secret.yaml":  file.modpath() + "datastore/staging/application-secret.yaml"}
}

[Peefy]

Hello @riven-blade

The file.modpath maybe cached cross different KCL modules, which is a bug. And I will fix it latter. Thanks for the feedback.