search

fluxcd / flux2

Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.
https://fluxcd.io
Apache License 2.0
6.48k stars 599 forks source link

Kustomization referencing manifest on GitHub fails to apply #3127

Open damoxc opened 2 years ago

damoxc commented 2 years ago

Describe the bug

Attempting to install the awx-operator using the Kustomize method, which references a manifest within the GitHub repository, this works fine when running kustomize build but committing it to the repository ready to be picked up by Flux results in:

✗ Kustomization reconciliation failed: kustomize build failed: accumulating resources: accumulation err='accumulating resources from './awx': read /tmp/kustomization-795774899/awx: is a directory': recursed accumulation of path '/tmp/kustomization-795774899/awx': accumulating resources: accumulation err='accumulating resources from 'github.com/ansible/awx-operator/config/default?ref=0.29.0': open /tmp/kustomization-795774899/awx/github.com/ansible/awx-operator/config/default?ref=0.29.0: no such file or directory': git cmd = '/usr/bin/git fetch --depth=1 origin 0.29.0': exit status 128

Steps to reproduce

  1. Include a kustomization.yaml which references a file in a GitHub repository 
    
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# Find the latest tag here: https://github.com/ansible/awx-operator/releases
- github.com/ansible/awx-operator/config/default?ref=<tag>

Set the image tags to match the git version from above

images:

Specify a custom namespace in which to install AWX

namespace: awx


2. Attempt to apply

### Expected behavior

Manifests would be fetched from GitHub, same behaviour as `kustomize build`.

### Screenshots and recordings

_No response_

### OS / Distro

Arch Linux

### Flux version

v0.34.0

### Flux check

► checking prerequisites
✔ Kubernetes 1.23.10+rke2r1 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.24.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.25.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.21.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.28.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.26.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.29.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

### Git provider

_No response_

### Container Registry provider

_No response_

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct
stefanprodan commented 2 years ago

Can you please try:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: awx-operator
  namespace: flux-system
spec:
  interval: 12h
  url: https://github.com/ansible/awx-operator
  ref:
    tag: 0.29.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: awx-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  path: ./config/default
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: awx-operator
  targetNamespace: awx
  images:
    - name: quay.io/ansible/awx-operator
       newTag: 0.29.0
damoxc commented 2 years ago

Thanks a lot, that works as expected.

Is it mentioned in the documentation that fetching manifests from URLs isn't supported in kustomize.yaml? I did try to spot whether or not it was.

stefanprodan commented 2 years ago

Is it mentioned in the documentation that fetching manifests from URLs isn't supported in kustomize.yaml?

We do support remote bases but we encourage users to disable it. In your case, I suspect that cloning fails because Kustomize itself does not retry, nor does it cache the repo like Flux does when you use a GitRepository.

From docs:

For security and performance reasons, it is advised to disallow the usage of remote bases in Kustomize overlays. To enforce this setting, platform admins can use the --no-remote-bases=true controller flag.

stefanprodan commented 2 years ago

It may work better if you tell Kustomize to clone the repo over HTTPS like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/ansible/awx-operator/config/default?ref=<tag>
polarroyo commented 1 year ago

I am experiencing the same behavior, but only when the kustomize-controller runs on a ARM (aarch64) node. When it runs on a x86 node I do not have any issue.

kustomize-controller version that I can reproduce the issue:

kifeo commented 1 year ago

I have the same issue even with https:// prefix :

 ---
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: monitoring
 resources:
   - https://github.com/fluxcd/flux2/manifests/monitoring/monitoring-config?ref=v0.39.0

log :

accumulating resources: accumulation err='accumulating resources from 'https://github.com/fluxcd/flux2/manifests/monitoring/monitoring-config?ref=v0.39.0': URL is a git repository': git cmd = '/usr/bin/git fetch --depth=1 origin v0.39.0': exit status 128

ilijamt commented 1 year ago

I have a similar issue with this, the only difference is I'm building the CRDs with flux.

This is my Kustomization to build all the CRDs I need. 

 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - github.com/ansible/awx-operator/config/crd?ref=1.1.3
   - external-secrets.io_clusterexternalsecrets.yaml
   - external-secrets.io_clustersecretstores.yaml
   - external-secrets.io_externalsecrets.yaml
   - external-secrets.io_secretstores.yaml
   - calico-crds.yaml
   - cert-manager.io_certificaterequests.yaml
   - cert-manager.io_certificates.yaml
   - cert-manager.io_challenges.acme.yaml
   - cert-manager.io_clusterissuers.yaml
   - cert-manager.io_issuers.yaml
   - cert-manager.io_orders.acme.yaml

When running both with version v4.5.7 and v5.1.1 this works and the CRDs for github.com/ansible/awx-operator/config/crd?ref=1.1.3 are present in the generated yaml. 

kustomize build <kustomization.path>

When running, which contains the kustomization above 

flux build kustomization crds --path crds

the AWX crds are missing from the generated yaml.

I've tried with both github.com/ansible/awx-operator/config/crd?ref=1.1.3 and https://github.com/ansible/awx-operator/config/crd?ref=1.1.3

With github.com/ansible/awx-operator/config/crd?ref=1.1.3 I just get an error 

✗ failed to generate kustomization.yaml: lstat /manifests/crds/github.com/ansible/awx-operator/config/crd/?ref=1.1.3: no such file or directory <nil> <nil>

and with https://github.com/ansible/awx-operator/config/crd?ref=1.1.3 the AWX CRDs are missing from the generated file.

$ flux check
► checking prerequisites
✔ Kubernetes 1.25.12-eks-2d98532 >=1.25.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.36.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.1.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.1.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.1.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed