Azure AKS: Flux v2.0.0-rc.1 fails with `gitrepositories Invalid value: "v1": must appear in spec.versions`

SachithKasthuriarachchi commented 1 year ago

Describe the bug

I installed the new flux-cli version and ran the flux bootstrap command to upgrade the components to the latest release candidate version. I get the following error in kustomize-controller.

CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io dry-run failed, reason: Invalid, error: CustomResourceDefinition.apiextensions.k8s.io "gitrepositories.source.toolkit.fluxcd.io" is invalid: status.storedVersions[0]: Invalid value: "v1": must appear in spec.versions

Steps to reproduce

curl -s https://fluxcd.io/install.sh | sudo bash
export GITHUB_TOKEN=<token>
flux bootstrap github --owner=<github org> --repository=<github repo> --path=<path> --branch=<branch>
Check the flux-system kustomization

Expected behavior

Should complete without any errors

Screenshots and recordings

No response

OS / Distro

Ubuntu

Flux version

v2.0.0-rc.1

Flux check

► checking prerequisites ✔ Kubernetes 1.24.9 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.32.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0-rc.1 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.0-rc.1 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

GitHub

Container Registry provider

No response

Additional context

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

stefanprodan commented 1 year ago

This looks like a bug in Kubernetes, since flux check does list v1 as stored in etcd gitrepositories.source.toolkit.fluxcd.io/v1. What Kubernetes distribution are you using?

As a workaround, you can run flux uninstall --keep-namespace then redoing bootstrap should work. Please make sure to wait for the CRDs to be deleted after uninstall, after kubectl get crds doesn't list the Flux CRDs only then you can rerun bootstrap. The uninstall command will not touch any workloads, so it's safe to run it.

SachithKasthuriarachchi commented 1 year ago

What Kubernetes distribution are you using?

@stefanprodan I am using Kubernetes 1.24.9

As a workaround, you can run flux uninstall --keep-namespace then redoing bootstrap should work

Still, I get the same error

► connecting to github.com
► cloning branch "main" from Git repository "<git repo>"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
✔ source secret up to date
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✗ client rate limiter Wait returned an error: context deadline exceeded
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
✗ bootstrap failed with 1 health check failure(s)

this timeouts eventually

stefanprodan commented 1 year ago

I was asking about distribution not version, is this EKS or GKE or AKS or some on-prem kops distribution?

After flux uninstall have you checked that GitRepository doesn't show up in kubectl get crds?

SachithKasthuriarachchi commented 1 year ago

I was asking about distribution not version, is this EKS or GKE or AKS or some on-prem kops distribution?

Ah sorry, this is AKS

After flux uninstall have you checked that GitRepository doesn't show up in kubectl get crds?

Yes, none of flux related CRDs showed up

stefanprodan commented 1 year ago

After bootstrap fails, can you please post here:

kubectl get crd gitrepositories.source.toolkit.fluxcd.io --show-managed-fields -oyaml

SachithKasthuriarachchi commented 1 year ago

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.1
  creationTimestamp: "2023-04-12T09:38:45Z"
  generation: 1
  labels:
    app.kubernetes.io/component: source-controller
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/version: v2.0.0-rc.1
  managedFields:
  - apiVersion: apiextensions.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:controller-gen.kubebuilder.io/version: {}
        f:labels:
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/part-of: {}
          f:app.kubernetes.io/version: {}
      f:spec:
        f:group: {}
        f:names:
          f:kind: {}
          f:listKind: {}
          f:plural: {}
          f:shortNames: {}
          f:singular: {}
        f:scope: {}
        f:versions: {}
    manager: flux
    operation: Apply
    time: "2023-04-12T09:38:45Z"
  - apiVersion: apiextensions.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind: {}
          f:listKind: {}
          f:plural: {}
          f:shortNames: {}
          f:singular: {}
        f:conditions:
          k:{"type":"Established"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"NamesAccepted"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: kube-apiserver
    operation: Update
    subresource: status
    time: "2023-04-12T09:38:45Z"
  name: gitrepositories.source.toolkit.fluxcd.io
  resourceVersion: "516022709"
  uid: d3c6bdaa-292a-4966-9eae-a7c8fa1b1dce
spec:
  conversion:
    strategy: None
  group: source.toolkit.fluxcd.io
  names:
    kind: GitRepository
    listKind: GitRepositoryList
    plural: gitrepositories
    shortNames:
    - gitrepo
    singular: gitrepository
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - jsonPath: .spec.url
      name: URL
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    - jsonPath: .status.conditions[?(@.type=="Ready")].status
      name: Ready
      type: string
    - jsonPath: .status.conditions[?(@.type=="Ready")].message
      name: Status
      type: string
    name: v1
    schema:
      openAPIV3Schema:
        description: GitRepository is the Schema for the gitrepositories API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: GitRepositorySpec specifies the required configuration to
              produce an Artifact for a Git repository.
            properties:
              ignore:
                description: Ignore overrides the set of excluded patterns in the
                  .sourceignore format (which is the same as .gitignore). If not provided,
                  a default will be used, consult the documentation for your version
                  to find out what those are.
                type: string
              include:
                description: Include specifies a list of GitRepository resources which
                  Artifacts should be included in the Artifact produced for this GitRepository.
                items:
                  description: GitRepositoryInclude specifies a local reference to
                    a GitRepository which Artifact (sub-)contents must be included,
                    and where they should be placed.
                  properties:
                    fromPath:
                      description: FromPath specifies the path to copy contents from,
                        defaults to the root of the Artifact.
                      type: string
                    repository:
                      description: GitRepositoryRef specifies the GitRepository which
                        Artifact contents must be included.
                      properties:
                        name:
                          description: Name of the referent.
                          type: string
                      required:
                      - name
                      type: object
                    toPath:
                      description: ToPath specifies the path to copy contents to,
                        defaults to the name of the GitRepositoryRef.
                      type: string
                  required:
                  - repository
                  type: object
                type: array
              interval:
                description: Interval at which to check the GitRepository for updates.
                pattern: ^([0-9]+(\.[0-9]+)?(ms|s|m|h))+$
                type: string
              recurseSubmodules:
                description: RecurseSubmodules enables the initialization of all submodules
                  within the GitRepository as cloned from the URL, using their default
                  settings.
                type: boolean
              ref:
                description: Reference specifies the Git reference to resolve and
                  monitor for changes, defaults to the 'master' branch.
                properties:
                  branch:
                    description: Branch to check out, defaults to 'master' if no other
                      field is defined.
                    type: string
                  commit:
                    description: "Commit SHA to check out, takes precedence over all
                      reference fields. \n This can be combined with Branch to shallow
                      clone the branch, in which the commit is expected to exist."
                    type: string
                  name:
                    description: "Name of the reference to check out; takes precedence
                      over Branch, Tag and SemVer. \n It must be a valid Git reference:
                      https://git-scm.com/docs/git-check-ref-format#_description Examples:
                      \"refs/heads/main\", \"refs/tags/v0.1.0\", \"refs/pull/420/head\",
                      \"refs/merge-requests/1/head\""
                    type: string
                  semver:
                    description: SemVer tag expression to check out, takes precedence
                      over Tag.
                    type: string
                  tag:
                    description: Tag to check out, takes precedence over Branch.
                    type: string
                type: object
              secretRef:
                description: SecretRef specifies the Secret containing authentication
                  credentials for the GitRepository. For HTTPS repositories the Secret
                  must contain 'username' and 'password' fields for basic auth or
                  'bearerToken' field for token auth. For SSH repositories the Secret
                  must contain 'identity' and 'known_hosts' fields.
                properties:
                  name:
                    description: Name of the referent.
                    type: string
                required:
                - name
                type: object
              suspend:
                description: Suspend tells the controller to suspend the reconciliation
                  of this GitRepository.
                type: boolean
              timeout:
                default: 60s
                description: Timeout for Git operations like cloning, defaults to
                  60s.
                pattern: ^([0-9]+(\.[0-9]+)?(ms|s|m))+$
                type: string
              url:
                description: URL specifies the Git repository URL, it can be an HTTP/S
                  or SSH address.
                pattern: ^(http|https|ssh)://.*$
                type: string
              verify:
                description: Verification specifies the configuration to verify the
                  Git commit signature(s).
                properties:
                  mode:
                    description: Mode specifies what Git object should be verified,
                      currently ('head').
                    enum:
                    - head
                    type: string
                  secretRef:
                    description: SecretRef specifies the Secret containing the public
                      keys of trusted Git authors.
                    properties:
                      name:
                        description: Name of the referent.
                        type: string
                    required:
                    - name
                    type: object
                required:
                - mode
                - secretRef
                type: object
            required:
            - interval
            - url
            type: object
          status:
            default:
              observedGeneration: -1
            description: GitRepositoryStatus records the observed state of a Git repository.
            properties:
              artifact:
                description: Artifact represents the last successful GitRepository
                  reconciliation.
                properties:
                  digest:
                    description: Digest is the digest of the file in the form of '<algorithm>:<checksum>'.
                    pattern: ^[a-z0-9]+(?:[.+_-][a-z0-9]+)*:[a-zA-Z0-9=_-]+$
                    type: string
                  lastUpdateTime:
                    description: LastUpdateTime is the timestamp corresponding to
                      the last update of the Artifact.
                    format: date-time
                    type: string
                  metadata:
                    additionalProperties:
                      type: string
                    description: Metadata holds upstream information such as OCI annotations.
                    type: object
                  path:
                    description: Path is the relative file path of the Artifact. It
                      can be used to locate the file in the root of the Artifact storage
                      on the local file system of the controller managing the Source.
                    type: string
                  revision:
                    description: Revision is a human-readable identifier traceable
                      in the origin source system. It can be a Git commit SHA, Git
                      tag, a Helm chart version, etc.
                    type: string
                  size:
                    description: Size is the number of bytes in the file.
                    format: int64
                    type: integer
                  url:
                    description: URL is the HTTP address of the Artifact as exposed
                      by the controller managing the Source. It can be used to retrieve
                      the Artifact for consumption, e.g. by another controller applying
                      the Artifact contents.
                    type: string
                required:
                - lastUpdateTime
                - path
                - revision
                - url
                type: object
              conditions:
                description: Conditions holds the conditions for the GitRepository.
                items:
                  description: "Condition contains details for one aspect of the current
                    state of this API Resource. --- This struct is intended for direct
                    use as an array at the field path .status.conditions.  For example,
                    \n type FooStatus struct{ // Represents the observations of a
                    foo's current state. // Known .status.conditions.type are: \"Available\",
                    \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge
                    // +listType=map // +listMapKey=type Conditions []metav1.Condition
                    `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\"
                    protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
                  properties:
                    lastTransitionTime:
                      description: lastTransitionTime is the last time the condition
                        transitioned from one status to another. This should be when
                        the underlying condition changed.  If that is not known, then
                        using the time when the API field changed is acceptable.
                      format: date-time
                      type: string
                    message:
                      description: message is a human readable message indicating
                        details about the transition. This may be an empty string.
                      maxLength: 32768
                      type: string
                    observedGeneration:
                      description: observedGeneration represents the .metadata.generation
                        that the condition was set based upon. For instance, if .metadata.generation
                        is currently 12, but the .status.conditions[x].observedGeneration
                        is 9, the condition is out of date with respect to the current
                        state of the instance.
                      format: int64
                      minimum: 0
                      type: integer
                    reason:
                      description: reason contains a programmatic identifier indicating
                        the reason for the condition's last transition. Producers
                        of specific condition types may define expected values and
                        meanings for this field, and whether the values are considered
                        a guaranteed API. The value should be a CamelCase string.
                        This field may not be empty.
                      maxLength: 1024
                      minLength: 1
                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      type: string
                    status:
                      description: status of the condition, one of True, False, Unknown.
                      enum:
                      - "True"
                      - "False"
                      - Unknown
                      type: string
                    type:
                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
                        --- Many .condition.type values are consistent across resources
                        like Available, but because arbitrary conditions can be useful
                        (see .node.status.conditions), the ability to deconflict is
                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      maxLength: 316
                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      type: string
                  required:
                  - lastTransitionTime
                  - message
                  - reason
                  - status
                  - type
                  type: object
                type: array
              includedArtifacts:
                description: IncludedArtifacts contains a list of the last successfully
                  included Artifacts as instructed by GitRepositorySpec.Include.
                items:
                  description: Artifact represents the output of a Source reconciliation.
                  properties:
                    digest:
                      description: Digest is the digest of the file in the form of
                        '<algorithm>:<checksum>'.
                      pattern: ^[a-z0-9]+(?:[.+_-][a-z0-9]+)*:[a-zA-Z0-9=_-]+$
                      type: string
                    lastUpdateTime:
                      description: LastUpdateTime is the timestamp corresponding to
                        the last update of the Artifact.
                      format: date-time
                      type: string
                    metadata:
                      additionalProperties:
                        type: string
                      description: Metadata holds upstream information such as OCI
                        annotations.
                      type: object
                    path:
                      description: Path is the relative file path of the Artifact.
                        It can be used to locate the file in the root of the Artifact
                        storage on the local file system of the controller managing
                        the Source.
                      type: string
                    revision:
                      description: Revision is a human-readable identifier traceable
                        in the origin source system. It can be a Git commit SHA, Git
                        tag, a Helm chart version, etc.
                      type: string
                    size:
                      description: Size is the number of bytes in the file.
                      format: int64
                      type: integer
                    url:
                      description: URL is the HTTP address of the Artifact as exposed
                        by the controller managing the Source. It can be used to retrieve
                        the Artifact for consumption, e.g. by another controller applying
                        the Artifact contents.
                      type: string
                  required:
                  - lastUpdateTime
                  - path
                  - revision
                  - url
                  type: object
                type: array
              lastHandledReconcileAt:
                description: LastHandledReconcileAt holds the value of the most recent
                  reconcile request value, so a change of the annotation value can
                  be detected.
                type: string
              observedGeneration:
                description: ObservedGeneration is the last observed generation of
                  the GitRepository object.
                format: int64
                type: integer
              observedIgnore:
                description: ObservedIgnore is the observed exclusion patterns used
                  for constructing the source artifact.
                type: string
              observedInclude:
                description: ObservedInclude is the observed list of GitRepository
                  resources used to produce the current Artifact.
                items:
                  description: GitRepositoryInclude specifies a local reference to
                    a GitRepository which Artifact (sub-)contents must be included,
                    and where they should be placed.
                  properties:
                    fromPath:
                      description: FromPath specifies the path to copy contents from,
                        defaults to the root of the Artifact.
                      type: string
                    repository:
                      description: GitRepositoryRef specifies the GitRepository which
                        Artifact contents must be included.
                      properties:
                        name:
                          description: Name of the referent.
                          type: string
                      required:
                      - name
                      type: object
                    toPath:
                      description: ToPath specifies the path to copy contents to,
                        defaults to the name of the GitRepositoryRef.
                      type: string
                  required:
                  - repository
                  type: object
                type: array
              observedRecurseSubmodules:
                description: ObservedRecurseSubmodules is the observed resource submodules
                  configuration used to produce the current Artifact.
                type: boolean
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
  - additionalPrinterColumns:
    - jsonPath: .spec.url
      name: URL
      type: string
    - jsonPath: .status.conditions[?(@.type=="Ready")].status
      name: Ready
      type: string
    - jsonPath: .status.conditions[?(@.type=="Ready")].message
      name: Status
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    deprecated: true
    deprecationWarning: v1beta1 GitRepository is deprecated, upgrade to v1
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: GitRepository is the Schema for the gitrepositories API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: GitRepositorySpec defines the desired state of a Git repository.
            properties:
              accessFrom:
                description: AccessFrom defines an Access Control List for allowing
                  cross-namespace references to this object.
                properties:
                  namespaceSelectors:
                    description: NamespaceSelectors is the list of namespace selectors
                      to which this ACL applies. Items in this list are evaluated
                      using a logical OR operation.
                    items:
                      description: NamespaceSelector selects the namespaces to which
                        this ACL applies. An empty map of MatchLabels matches all
                        namespaces in a cluster.
                      properties:
                        matchLabels:
                          additionalProperties:
                            type: string
                          description: MatchLabels is a map of {key,value} pairs.
                            A single {key,value} in the matchLabels map is equivalent
                            to an element of matchExpressions, whose key field is
                            "key", the operator is "In", and the values array contains
                            only "value". The requirements are ANDed.
                          type: object
                      type: object
                    type: array
                required:
                - namespaceSelectors
                type: object
              gitImplementation:
                default: go-git
                description: Determines which git client library to use. Defaults
                  to go-git, valid values are ('go-git', 'libgit2').
                enum:
                - go-git
                - libgit2
                type: string
              ignore:
                description: Ignore overrides the set of excluded patterns in the
                  .sourceignore format (which is the same as .gitignore). If not provided,
                  a default will be used, consult the documentation for your version
                  to find out what those are.
                type: string
              include:
                description: Extra git repositories to map into the repository
                items:
                  description: GitRepositoryInclude defines a source with a from and
                    to path.
                  properties:
                    fromPath:
                      description: The path to copy contents from, defaults to the
                        root directory.
                      type: string
                    repository:
                      description: Reference to a GitRepository to include.
                      properties:
                        name:
                          description: Name of the referent.
                          type: string
                      required:
                      - name
                      type: object
                    toPath:
                      description: The path to copy contents to, defaults to the name
                        of the source ref.
                      type: string
                  required:
                  - repository
                  type: object
                type: array
              interval:
                description: The interval at which to check for repository updates.
                type: string
              recurseSubmodules:
                description: When enabled, after the clone is created, initializes
                  all submodules within, using their default settings. This option
                  is available only when using the 'go-git' GitImplementation.
                type: boolean
              ref:
                description: The Git reference to checkout and monitor for changes,
                  defaults to master branch.
                properties:
                  branch:
                    description: The Git branch to checkout, defaults to master.
                    type: string
                  commit:
                    description: The Git commit SHA to checkout, if specified Tag
                      filters will be ignored.
                    type: string
                  semver:
                    description: The Git tag semver expression, takes precedence over
                      Tag.
                    type: string
                  tag:
                    description: The Git tag to checkout, takes precedence over Branch.
                    type: string
                type: object
              secretRef:
                description: The secret name containing the Git credentials. For HTTPS
                  repositories the secret must contain username and password fields.
                  For SSH repositories the secret must contain identity and known_hosts
                  fields.
                properties:
                  name:
                    description: Name of the referent.
                    type: string
                required:
                - name
                type: object
              suspend:
                description: This flag tells the controller to suspend the reconciliation
                  of this source.
                type: boolean
              timeout:
                default: 60s
                description: The timeout for remote Git operations like cloning, defaults
                  to 60s.
                type: string
              url:
                description: The repository URL, can be a HTTP/S or SSH address.
                pattern: ^(http|https|ssh)://.*$
                type: string
              verify:
                description: Verify OpenPGP signature for the Git commit HEAD points
                  to.
                properties:
                  mode:
                    description: Mode describes what git object should be verified,
                      currently ('head').
                    enum:
                    - head
                    type: string
                  secretRef:
                    description: The secret name containing the public keys of all
                      trusted Git authors.
                    properties:
                      name:
                        description: Name of the referent.
                        type: string
                    required:
                    - name
                    type: object
                required:
                - mode
                type: object
            required:
            - interval
            - url
            type: object
          status:
            default:
              observedGeneration: -1
            description: GitRepositoryStatus defines the observed state of a Git repository.
            properties:
              artifact:
                description: Artifact represents the output of the last successful
                  repository sync.
                properties:
                  checksum:
                    description: Checksum is the SHA256 checksum of the artifact.
                    type: string
                  lastUpdateTime:
                    description: LastUpdateTime is the timestamp corresponding to
                      the last update of this artifact.
                    format: date-time
                    type: string
                  path:
                    description: Path is the relative file path of this artifact.
                    type: string
                  revision:
                    description: Revision is a human readable identifier traceable
                      in the origin source system. It can be a Git commit SHA, Git
                      tag, a Helm index timestamp, a Helm chart version, etc.
                    type: string
                  url:
                    description: URL is the HTTP address of this artifact.
                    type: string
                required:
                - path
                - url
                type: object
              conditions:
                description: Conditions holds the conditions for the GitRepository.
                items:
                  description: "Condition contains details for one aspect of the current
                    state of this API Resource. --- This struct is intended for direct
                    use as an array at the field path .status.conditions.  For example,
                    \n type FooStatus struct{ // Represents the observations of a
                    foo's current state. // Known .status.conditions.type are: \"Available\",
                    \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge
                    // +listType=map // +listMapKey=type Conditions []metav1.Condition
                    `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\"
                    protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
                  properties:
                    lastTransitionTime:
                      description: lastTransitionTime is the last time the condition
                        transitioned from one status to another. This should be when
                        the underlying condition changed.  If that is not known, then
                        using the time when the API field changed is acceptable.
                      format: date-time
                      type: string
                    message:
                      description: message is a human readable message indicating
                        details about the transition. This may be an empty string.
                      maxLength: 32768
                      type: string
                    observedGeneration:
                      description: observedGeneration represents the .metadata.generation
                        that the condition was set based upon. For instance, if .metadata.generation
                        is currently 12, but the .status.conditions[x].observedGeneration
                        is 9, the condition is out of date with respect to the current
                        state of the instance.
                      format: int64
                      minimum: 0
                      type: integer
                    reason:
                      description: reason contains a programmatic identifier indicating
                        the reason for the condition's last transition. Producers
                        of specific condition types may define expected values and
                        meanings for this field, and whether the values are considered
                        a guaranteed API. The value should be a CamelCase string.
                        This field may not be empty.
                      maxLength: 1024
                      minLength: 1
                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      type: string
                    status:
                      description: status of the condition, one of True, False, Unknown.
                      enum:
                      - "True"
                      - "False"
                      - Unknown
                      type: string
                    type:
                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
                        --- Many .condition.type values are consistent across resources
                        like Available, but because arbitrary conditions can be useful
                        (see .node.status.conditions), the ability to deconflict is
                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      maxLength: 316
                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      type: string
                  required:
                  - lastTransitionTime
                  - message
                  - reason
                  - status
                  - type
                  type: object
                type: array
              includedArtifacts:
                description: IncludedArtifacts represents the included artifacts from
                  the last successful repository sync.
                items:
                  description: Artifact represents the output of a source synchronisation.
                  properties:
                    checksum:
                      description: Checksum is the SHA256 checksum of the artifact.
                      type: string
                    lastUpdateTime:
                      description: LastUpdateTime is the timestamp corresponding to
                        the last update of this artifact.
                      format: date-time
                      type: string
                    path:
                      description: Path is the relative file path of this artifact.
                      type: string
                    revision:
                      description: Revision is a human readable identifier traceable
                        in the origin source system. It can be a Git commit SHA, Git
                        tag, a Helm index timestamp, a Helm chart version, etc.
                      type: string
                    url:
                      description: URL is the HTTP address of this artifact.
                      type: string
                  required:
                  - path
                  - url
                  type: object
                type: array
              lastHandledReconcileAt:
                description: LastHandledReconcileAt holds the value of the most recent
                  reconcile request value, so a change of the annotation value can
                  be detected.
                type: string
              observedGeneration:
                description: ObservedGeneration is the last observed generation.
                format: int64
                type: integer
              url:
                description: URL is the download link for the artifact output of the
                  last repository sync.
                type: string
            type: object
        type: object
    served: true
    storage: false
    subresources:
      status: {}
  - additionalPrinterColumns:
    - jsonPath: .spec.url
      name: URL
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    - jsonPath: .status.conditions[?(@.type=="Ready")].status
      name: Ready
      type: string
    - jsonPath: .status.conditions[?(@.type=="Ready")].message
      name: Status
      type: string
    deprecated: true
    deprecationWarning: v1beta2 GitRepository is deprecated, upgrade to v1
    name: v1beta2
    schema:
      openAPIV3Schema:
        description: GitRepository is the Schema for the gitrepositories API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: GitRepositorySpec specifies the required configuration to
              produce an Artifact for a Git repository.
            properties:
              accessFrom:
                description: 'AccessFrom specifies an Access Control List for allowing
                  cross-namespace references to this object. NOTE: Not implemented,
                  provisional as of https://github.com/fluxcd/flux2/pull/2092'
                properties:
                  namespaceSelectors:
                    description: NamespaceSelectors is the list of namespace selectors
                      to which this ACL applies. Items in this list are evaluated
                      using a logical OR operation.
                    items:
                      description: NamespaceSelector selects the namespaces to which
                        this ACL applies. An empty map of MatchLabels matches all
                        namespaces in a cluster.
                      properties:
                        matchLabels:
                          additionalProperties:
                            type: string
                          description: MatchLabels is a map of {key,value} pairs.
                            A single {key,value} in the matchLabels map is equivalent
                            to an element of matchExpressions, whose key field is
                            "key", the operator is "In", and the values array contains
                            only "value". The requirements are ANDed.
                          type: object
                      type: object
                    type: array
                required:
                - namespaceSelectors
                type: object
              gitImplementation:
                default: go-git
                description: 'GitImplementation specifies which Git client library
                  implementation to use. Defaults to ''go-git'', valid values are
                  (''go-git'', ''libgit2''). Deprecated: gitImplementation is deprecated
                  now that ''go-git'' is the only supported implementation.'
                enum:
                - go-git
                - libgit2
                type: string
              ignore:
                description: Ignore overrides the set of excluded patterns in the
                  .sourceignore format (which is the same as .gitignore). If not provided,
                  a default will be used, consult the documentation for your version
                  to find out what those are.
                type: string
              include:
                description: Include specifies a list of GitRepository resources which
                  Artifacts should be included in the Artifact produced for this GitRepository.
                items:
                  description: GitRepositoryInclude specifies a local reference to
                    a GitRepository which Artifact (sub-)contents must be included,
                    and where they should be placed.
                  properties:
                    fromPath:
                      description: FromPath specifies the path to copy contents from,
                        defaults to the root of the Artifact.
                      type: string
                    repository:
                      description: GitRepositoryRef specifies the GitRepository which
                        Artifact contents must be included.
                      properties:
                        name:
                          description: Name of the referent.
                          type: string
                      required:
                      - name
                      type: object
                    toPath:
                      description: ToPath specifies the path to copy contents to,
                        defaults to the name of the GitRepositoryRef.
                      type: string
                  required:
                  - repository
                  type: object
                type: array
              interval:
                description: Interval at which to check the GitRepository for updates.
                pattern: ^([0-9]+(\.[0-9]+)?(ms|s|m|h))+$
                type: string
              recurseSubmodules:
                description: RecurseSubmodules enables the initialization of all submodules
                  within the GitRepository as cloned from the URL, using their default
                  settings.
                type: boolean
              ref:
                description: Reference specifies the Git reference to resolve and
                  monitor for changes, defaults to the 'master' branch.
                properties:
                  branch:
                    description: Branch to check out, defaults to 'master' if no other
                      field is defined.
                    type: string
                  commit:
                    description: "Commit SHA to check out, takes precedence over all
                      reference fields. \n This can be combined with Branch to shallow
                      clone the branch, in which the commit is expected to exist."
                    type: string
                  name:
                    description: "Name of the reference to check out; takes precedence
                      over Branch, Tag and SemVer. \n It must be a valid Git reference:
                      https://git-scm.com/docs/git-check-ref-format#_description Examples:
                      \"refs/heads/main\", \"refs/tags/v0.1.0\", \"refs/pull/420/head\",
                      \"refs/merge-requests/1/head\""
                    type: string
                  semver:
                    description: SemVer tag expression to check out, takes precedence
                      over Tag.
                    type: string
                  tag:
                    description: Tag to check out, takes precedence over Branch.
                    type: string
                type: object
              secretRef:
                description: SecretRef specifies the Secret containing authentication
                  credentials for the GitRepository. For HTTPS repositories the Secret
                  must contain 'username' and 'password' fields for basic auth or
                  'bearerToken' field for token auth. For SSH repositories the Secret
                  must contain 'identity' and 'known_hosts' fields.
                properties:
                  name:
                    description: Name of the referent.
                    type: string
                required:
                - name
                type: object
              suspend:
                description: Suspend tells the controller to suspend the reconciliation
                  of this GitRepository.
                type: boolean
              timeout:
                default: 60s
                description: Timeout for Git operations like cloning, defaults to
                  60s.
                pattern: ^([0-9]+(\.[0-9]+)?(ms|s|m))+$
                type: string
              url:
                description: URL specifies the Git repository URL, it can be an HTTP/S
                  or SSH address.
                pattern: ^(http|https|ssh)://.*$
                type: string
              verify:
                description: Verification specifies the configuration to verify the
                  Git commit signature(s).
                properties:
                  mode:
                    description: Mode specifies what Git object should be verified,
                      currently ('head').
                    enum:
                    - head
                    type: string
                  secretRef:
                    description: SecretRef specifies the Secret containing the public
                      keys of trusted Git authors.
                    properties:
                      name:
                        description: Name of the referent.
                        type: string
                    required:
                    - name
                    type: object
                required:
                - mode
                - secretRef
                type: object
            required:
            - interval
            - url
            type: object
          status:
            default:
              observedGeneration: -1
            description: GitRepositoryStatus records the observed state of a Git repository.
            properties:
              artifact:
                description: Artifact represents the last successful GitRepository
                  reconciliation.
                properties:
                  digest:
                    description: Digest is the digest of the file in the form of '<algorithm>:<checksum>'.
                    pattern: ^[a-z0-9]+(?:[.+_-][a-z0-9]+)*:[a-zA-Z0-9=_-]+$
                    type: string
                  lastUpdateTime:
                    description: LastUpdateTime is the timestamp corresponding to
                      the last update of the Artifact.
                    format: date-time
                    type: string
                  metadata:
                    additionalProperties:
                      type: string
                    description: Metadata holds upstream information such as OCI annotations.
                    type: object
                  path:
                    description: Path is the relative file path of the Artifact. It
                      can be used to locate the file in the root of the Artifact storage
                      on the local file system of the controller managing the Source.
                    type: string
                  revision:
                    description: Revision is a human-readable identifier traceable
                      in the origin source system. It can be a Git commit SHA, Git
                      tag, a Helm chart version, etc.
                    type: string
                  size:
                    description: Size is the number of bytes in the file.
                    format: int64
                    type: integer
                  url:
                    description: URL is the HTTP address of the Artifact as exposed
                      by the controller managing the Source. It can be used to retrieve
                      the Artifact for consumption, e.g. by another controller applying
                      the Artifact contents.
                    type: string
                required:
                - lastUpdateTime
                - path
                - revision
                - url
                type: object
              conditions:
                description: Conditions holds the conditions for the GitRepository.
                items:
                  description: "Condition contains details for one aspect of the current
                    state of this API Resource. --- This struct is intended for direct
                    use as an array at the field path .status.conditions.  For example,
                    \n type FooStatus struct{ // Represents the observations of a
                    foo's current state. // Known .status.conditions.type are: \"Available\",
                    \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge
                    // +listType=map // +listMapKey=type Conditions []metav1.Condition
                    `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\"
                    protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
                  properties:
                    lastTransitionTime:
                      description: lastTransitionTime is the last time the condition
                        transitioned from one status to another. This should be when
                        the underlying condition changed.  If that is not known, then
                        using the time when the API field changed is acceptable.
                      format: date-time
                      type: string
                    message:
                      description: message is a human readable message indicating
                        details about the transition. This may be an empty string.
                      maxLength: 32768
                      type: string
                    observedGeneration:
                      description: observedGeneration represents the .metadata.generation
                        that the condition was set based upon. For instance, if .metadata.generation
                        is currently 12, but the .status.conditions[x].observedGeneration
                        is 9, the condition is out of date with respect to the current
                        state of the instance.
                      format: int64
                      minimum: 0
                      type: integer
                    reason:
                      description: reason contains a programmatic identifier indicating
                        the reason for the condition's last transition. Producers
                        of specific condition types may define expected values and
                        meanings for this field, and whether the values are considered
                        a guaranteed API. The value should be a CamelCase string.
                        This field may not be empty.
                      maxLength: 1024
                      minLength: 1
                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      type: string
                    status:
                      description: status of the condition, one of True, False, Unknown.
                      enum:
                      - "True"
                      - "False"
                      - Unknown
                      type: string
                    type:
                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
                        --- Many .condition.type values are consistent across resources
                        like Available, but because arbitrary conditions can be useful
                        (see .node.status.conditions), the ability to deconflict is
                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      maxLength: 316
                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      type: string
                  required:
                  - lastTransitionTime
                  - message
                  - reason
                  - status
                  - type
                  type: object
                type: array
              contentConfigChecksum:
                description: "ContentConfigChecksum is a checksum of all the configurations
                  related to the content of the source artifact: - .spec.ignore -
                  .spec.recurseSubmodules - .spec.included and the checksum of the
                  included artifacts observed in .status.observedGeneration version
                  of the object. This can be used to determine if the content of the
                  included repository has changed. It has the format of `<algo>:<checksum>`,
                  for example: `sha256:<checksum>`. \n Deprecated: Replaced with explicit
                  fields for observed artifact content config in the status."
                type: string
              includedArtifacts:
                description: IncludedArtifacts contains a list of the last successfully
                  included Artifacts as instructed by GitRepositorySpec.Include.
                items:
                  description: Artifact represents the output of a Source reconciliation.
                  properties:
                    digest:
                      description: Digest is the digest of the file in the form of
                        '<algorithm>:<checksum>'.
                      pattern: ^[a-z0-9]+(?:[.+_-][a-z0-9]+)*:[a-zA-Z0-9=_-]+$
                      type: string
                    lastUpdateTime:
                      description: LastUpdateTime is the timestamp corresponding to
                        the last update of the Artifact.
                      format: date-time
                      type: string
                    metadata:
                      additionalProperties:
                        type: string
                      description: Metadata holds upstream information such as OCI
                        annotations.
                      type: object
                    path:
                      description: Path is the relative file path of the Artifact.
                        It can be used to locate the file in the root of the Artifact
                        storage on the local file system of the controller managing
                        the Source.
                      type: string
                    revision:
                      description: Revision is a human-readable identifier traceable
                        in the origin source system. It can be a Git commit SHA, Git
                        tag, a Helm chart version, etc.
                      type: string
                    size:
                      description: Size is the number of bytes in the file.
                      format: int64
                      type: integer
                    url:
                      description: URL is the HTTP address of the Artifact as exposed
                        by the controller managing the Source. It can be used to retrieve
                        the Artifact for consumption, e.g. by another controller applying
                        the Artifact contents.
                      type: string
                  required:
                  - lastUpdateTime
                  - path
                  - revision
                  - url
                  type: object
                type: array
              lastHandledReconcileAt:
                description: LastHandledReconcileAt holds the value of the most recent
                  reconcile request value, so a change of the annotation value can
                  be detected.
                type: string
              observedGeneration:
                description: ObservedGeneration is the last observed generation of
                  the GitRepository object.
                format: int64
                type: integer
              observedIgnore:
                description: ObservedIgnore is the observed exclusion patterns used
                  for constructing the source artifact.
                type: string
              observedInclude:
                description: ObservedInclude is the observed list of GitRepository
                  resources used to to produce the current Artifact.
                items:
                  description: GitRepositoryInclude specifies a local reference to
                    a GitRepository which Artifact (sub-)contents must be included,
                    and where they should be placed.
                  properties:
                    fromPath:
                      description: FromPath specifies the path to copy contents from,
                        defaults to the root of the Artifact.
                      type: string
                    repository:
                      description: GitRepositoryRef specifies the GitRepository which
                        Artifact contents must be included.
                      properties:
                        name:
                          description: Name of the referent.
                          type: string
                      required:
                      - name
                      type: object
                    toPath:
                      description: ToPath specifies the path to copy contents to,
                        defaults to the name of the GitRepositoryRef.
                      type: string
                  required:
                  - repository
                  type: object
                type: array
              observedRecurseSubmodules:
                description: ObservedRecurseSubmodules is the observed resource submodules
                  configuration used to produce the current Artifact.
                type: boolean
              url:
                description: URL is the dynamic fetch link for the latest Artifact.
                  It is provided on a "best effort" basis, and using the precise GitRepositoryStatus.Artifact
                  data is recommended.
                type: string
            type: object
        type: object
    served: true
    storage: false
    subresources:
      status: {}
status:
  acceptedNames:
    kind: GitRepository
    listKind: GitRepositoryList
    plural: gitrepositories
    shortNames:
    - gitrepo
    singular: gitrepository
  conditions:
  - lastTransitionTime: "2023-04-12T09:38:45Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2023-04-12T09:38:45Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1

stefanprodan commented 1 year ago

This looks Ok, v1 is registered and it also has the right spec, I can't explain Invalid value: "v1": must appear in spec.versions. Please reach out to Azure support as this must be an AKS bug.

SachithKasthuriarachchi commented 1 year ago

hi @stefanprodan I believe this is not specific to the AKS due to the following reason.

flux-system manifests was stored in a git repository. When I ran flux bootstrap again to upgrade it to the 2.0.0 rc I got the above error. Uninstalling and reinstalling didn't work either.

Now I created a new git repository and pointed the flux bootstrap to that new repository after uninstalling the existing flux-system components. This time it completed successfully. Therefore, my gut feeling is some glitch should be there in how the upgrades are handled in flux

Additional findings

https://github.com/fluxcd/flux2/issues/3362 https://github.com/fluxcd/flux2/issues/2743

stefanprodan commented 1 year ago

Can you please post here the gotk-sync.yaml also did you made any changes to Flux in kustomization.yaml? We've tested the upgrade of existing repos and clusters and we haven't encountered this issue, so we need help replicating this. cc @aryan9600

SachithKasthuriarachchi commented 1 year ago

here is the gotk-sync.yaml

# This manifest was generated by flux. DO NOT EDIT.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: ssh://git@github.com/SachithKasthuriarachchi/flux-experimental3
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./kustomize
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

to provide you with more context, here is how I first installed flux in the cluster.

Create flux-system namespace manually in the cluster
Generate an SSH key pair and deploy the private key as a kubernetes secret. No passphrase should be given when generating the SSH keypair. The output is assumed to be ‘identity’ and ‘identity.pub’ in the following commands

ssh-keygen -t ecdsa-sha2-nistp256

kubectl create secret generic -n flux-system flux-system --from-file ./identity --from-literal=known_hosts="github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="

Remove the private key after creating the flux-system secret
Add the content of ‘identity.pub’ as a deploy key in the github repository (no need to provide the write-access when adding the deploy key)
Commit the flux-system components in the github repository in a folder named as flux-components
Create a kustomize overlay in the git repo to include the flux-components overlay
Applied kubectl apply manually for the first time (it succeeded and git repo started to sync)

Then, when I'm upgrading I used the flux-cli which created an additional directory called flux-system in my repository. Therefore, I removed the previous flux-components overlay from the kustomization.yaml and included the flux-system directory. That's how I did the upgrade.

So the error is due to me doing it in the manual way first?

stefanprodan commented 1 year ago

Many things look bad in your example, please start fresh and follow the install docs. The repo structure should be clusters/my-aks/ this path should be given to bootstrap not ./kustomize.

To start fresh, uninstall flux and also delete the namespace. Then run bootstrap like:

flux bootstrap github --path clusters/my-aks

SachithKasthuriarachchi commented 1 year ago

@stefanprodan the above approach was followed since the generated PAT has access to all the repos in a given github organization. Further, write access to the repository should not be needed when the repository already contains the flux-system manifests.

The problem with a PAT is that anyone who has access to the PAT can clone all of the private repos in a github organization. The deploy key on the other hand has access only to the single repository.

So what I assume is, this should have been worked correctly if I updated the flux manifests in the git repo and let the flux do the sync instead of running the flux bootstrap. Please correct me if I am wrong.

stefanprodan commented 1 year ago

flux bootstrap github does not stores the PAT on the cluster, for each cluster we generate a SSH deploy key with read-only access, please see the docs https://fluxcd.io/flux/installation/

SachithKasthuriarachchi commented 1 year ago

flux bootstrap github does not stores the PAT on the cluster, for each cluster we generate a SSH deploy key with read-only access, please see the docs https://fluxcd.io/flux/installation/

oh great. Of course it is just an environment variable needed when we run the bootstrap command. Thanks for the clarification. I am closing this issue since it is resolved. Thanks a lot @stefanprodan and the team for introducing such a great tool!

fluxcd / flux2