feat!: central cache with symlinks

Zebradil commented 6 months ago

This PR is built on top of #257 and changes it to use symlinks for maintaining the central cache. Tests pass but I might have missed something (also, there are no new tests for new functions, it's TBD for later).

TODO:

[x] cleanup old links from the vendor directory
[x] exclude content's path from cache name generation

Create tasks to:

[x] cleanup cache storage from old downloads (only possible when running myks sync ALL) (#281)
[x] fix race condition when writing to cache (look for a comment in the code) (#283)

fritzduchardt commented 6 months ago

Hi @Zebradil, I would like to talk about this one in a call. To give you a head start:

Does not render fully:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: charts/
  contents:
  - path: cert-manager
    lazy: true
    helmChart:
      name: cert-manager
      version: v1.14.2
      repository:
        url: https://charts.jetstack.io
- path: charts/
  contents:
  - path: multus-cni
    lazy: true
    helmChart:
      name: multus-cni
      version: 1.1.10
      repository:
        url: https://charts.bitnami.com/bitnami

Does not work at all:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: charts/
  contents:
  - path: cert-manager
    lazy: true
    helmChart:
      name: cert-manager
      version: v1.14.2
      repository:
        url: https://charts.jetstack.io
  - path: multus-cni
    lazy: true
    helmChart:
      name: multus-cni
      version: 1.1.10
      repository:
        url: https://charts.bitnami.com/bitnami

Zebradil commented 6 months ago

Thank you for the examples. I'll take a closer look later today. But right now I can tell that the second example doesn't work because I explicitly restrict vendir configs to have only a single content per directory.

Zebradil commented 6 months ago

@fritzduchardt The first example is not valid vendir config:

$ cat <<EOF | vendir sync -f-
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: charts/
    contents:
      - path: cert-manager
        lazy: true
        helmChart:
          name: cert-manager
          version: v1.14.2
          repository:
            url: https://charts.jetstack.io
  - path: charts/
    contents:
      - path: multus-cni
        lazy: true
        helmChart:
          name: multus-cni
          version: 1.1.10
          repository:
            url: https://charts.bitnami.com/bitnami
EOF
vendir: Error: Parsing resource config '-':
  Unmarshaling config:
    Validating config:
      Expected to not manage overlapping paths: 'charts/' and 'charts/'

If I change the config in the following way, sync and render work correctly:

--- orig.yaml   2024-05-01 15:36:33.513465042 +0200
+++ changed.yaml    2024-05-01 15:36:07.393463391 +0200
@@ -1,18 +1,18 @@
 apiVersion: vendir.k14s.io/v1alpha1
 kind: Config
 directories:
-  - path: charts/
+  - path: charts/cert-manager
     contents:
-      - path: cert-manager
+      - path: .
         lazy: true
         helmChart:
           name: cert-manager
           version: v1.14.2
           repository:
             url: https://charts.jetstack.io
-  - path: charts/
+  - path: charts/multus-cni
     contents:
-      - path: multus-cni
+      - path: .
         lazy: true
         helmChart:
           name: multus-cni

We can implement support for that, but I'd approach this as a separate topic. What we should definitely have is a proper error message in this case.

fritzduchardt commented 6 months ago

Thanks for fixing the second example. However, rendering with this only leads to the first chart being rendered for me.

Zebradil commented 6 months ago

Thanks for fixing the second example

But I didn't fix it. The second example won't work and it was my deliberate decision. I mentioned this in a comment earlier: https://github.com/mykso/myks/pull/274#issuecomment-2088285621

I also briefly touched on that during our last call, but that probably was missed among lots of other details we discussed.

The reason for that was some logical complications of that state of the code, it could be already not an issue, I'll double-check that.

However, rendering with this only leads to the first chart being rendered for me.

Sorry, I didn't get. With wich one? Could you post the config which doesn't work for you?

fritzduchardt commented 6 months ago

My bad - I meant the first example, the one you fixed. That one only renders the first workload.

Zebradil commented 6 months ago

@fritzduchardt sorry, but I can't reproduce.

With this vendir config (your first example, fixed): https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/prototypes/two-charts/vendir/all.yaml
Cert-manager is rendered: https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/rendered/envs/mykso-dev/two-charts/deployment-cert-manager.yaml
Multus-cni is rendered: https://github.com/Zebradil/vigilant-bassoon/blob/a058bd96be8dd1ff4269ec62ddb6ca7e285a5340/rendered/envs/mykso-dev/two-charts/daemonset-multus-cni.yaml

Do I miss something?

fritzduchardt commented 6 months ago

Damn, I keep testing vendir.yamls that have overlapping paths (I put the dot at directory path level), which is now properly validated by vendir. However, this is not validated by myks, so it tries to sync and render those invalid vendir.yamls, without feedback to the user leading to unexpected results. I overlooked your comment regarding this a while back, sorry. Let's get this one merged, but put in a follow-up to ensure that we don't lose vendir validation.

mykso / myks

feat!: central cache with symlinks #274