Auto-Deploying Manifests ignores symlinked directories

frederictobiasc commented 4 months ago

Environmental Info: K3s Version: k3s version v1.26.6+k3s1 (3b1919b0) go version go1.20.8

Node(s) CPU architecture, OS, and Version: Linux 6.1.59 #1-NixOS SMP PREEMPT_DYNAMIC Thu Oct 19 21:08:58 UTC 2023 x86_64 GNU/Linux

Cluster Configuration: single-node test

Describe the bug: Auto-Deploying Manifests feature does not recognize manifests located in directory behind symlink.

Steps To Reproduce:

mkdir /var/lib/foo
cat <<EOF > /var/lib/foo/bar.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: bar
EOF
ln -s /var/lib/foo /var/lib/rancher/k3s/server/manifests

Expected behavior: k3s kubectl get ns contains namespace bar

Actual behavior: k3s kubectl get ns does not contain namespace bar

Additional context / logs:

Logs don't show anything
Copying the bar.yaml file into a regular directory in /var/lib/rancher/k3s/server/manifests works as expected.

brandond commented 4 months ago

The deploy controller uses filepath.Walk to list files, which does not follow symlinks.

https://github.com/k3s-io/k3s/blob/6d77b7a9204ebe40c53425ce4bc82c1df456e911/pkg/deploy/controller.go#L118

Can you explain your use case for wanting to keep your manifests outside the manifests dir?

frederictobiasc commented 4 months ago

Hi, thanks for your quick reply!

I'm using k3s in NixOS.

In my case, I build my k3s configuration with Nix. All NixOS build results are placed in the Nix store (/nix/store/). For an application to use its configuration, Nix creates symlinks in specific configuration directories that point to the actual configuration files in the Nix store.

However, while other methods exist for supplying configuration to applications in Nix, such as passing the configuration's path (inside /nix/store) as an argument, to the application, to my understanding, the Auto-Deploying Manifests feature supports only the discovery via the search path.

Therefore, as a Nix user, it would be greatly beneficial if k3s could either recognize symbolic links for manifest discovery or offer a mechanism to incorporate additional search paths for manifests.

endawkins commented 1 week ago

Validated on branch master with commit df5db28 / version 1.30

Environment Details

Infrastructure

[X] Cloud
[ ] Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-15-31 5.15.0-1019-aws #23-Ubuntu SMP Wed Aug 17 18:33:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

single node

Config.yaml:

write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
cluster-init: true
node-name: ip-[redacted].us-east-2.compute.internal
protect-kernel-defaults: true
selinux: true
node-external-ip: [redacted]
node-ip: [redacted]
secrets-encryption: true
kube-apiserver-arg:
  - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  - 'audit-log-maxage=30'
  - 'audit-log-maxbackup=10'
  - 'audit-log-maxsize=100'
  - 'request-timeout=300s'
  - 'service-account-lookup=true'
  - 'enable-admission-plugins=NodeRestriction,PodSecurity,NamespaceLifecycle,ServiceAccount'
  - 'admission-control-config-file=/var/lib/rancher/k3s/server/cluster-level-pss.yaml'
kube-controller-manager-arg:
  - 'terminated-pod-gc-threshold=10'
  - 'use-service-account-credentials=true'
kubelet-arg:
  - 'streaming-connection-idle-timeout=5m'
  - 'make-iptables-util-chains=true'

Additional files

bar.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: bar

Testing Steps

Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

Install k3s

$ sudo -i
$ mkdir /var/lib/foo/
$ cat <<EOF > /var/lib/foo/bar.yaml
$ > apiVersion: v1
kind: Namespace
metadata:
name: bar
$ > EOF
$ ln -s /var/lib/foo /var/lib/rancher/k3s/server/manifests
$ exit

Check for bar namespace: k3s kubectl get ns

Replication Results:

k3s version used for replication:

k3s -v
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

k3s kubectl get ns
NAME                 STATUS   AGE
default              Active   26m
kube-node-lease      Active   26m
kube-public          Active   26m
kube-system          Active   27m
local-path-storage   Active   23m

Validation Results:

k3s version used for validation:

k3s -v
k3s version v1.30.1+k3s-df5db28a (df5db28a)
go version go1.22.2

k3s kubectl get ns
NAME                 STATUS   AGE
bar                  Active   2s
default              Active   23m
kube-node-lease      Active   23m
kube-public          Active   23m
kube-system          Active   23m
local-path-storage   Active   17m

Additional context / logs:

N/A

k3s-io / k3s