cannot issue any auto certificates, although manual cert-manager certs work ok

The pod log is filled with:

E0915 09:58:20.754866       1 reflector.go:138] k8s.io/client-go@v0.20.7/tools/cache/reflector.go:167: Failed to watch *v1.Challenge: failed to list *v1.Challenge: the server could not find the requested resource (get challenges.acme.cert-manager.io)
E0915 09:58:30.442328       1 reflector.go:138] k8s.io/client-go@v0.20.7/tools/cache/reflector.go:167: Failed to watch *v1.Certificate: failed to list *v1.Certificate: the server could not find the requested resource (get certificates.cert-manager.io)
E0915 09:58:42.565343       1 reflector.go:138] k8s.io/client-go@v0.20.7/tools/cache/reflector.go:167: Failed to watch *v1.ClusterIssuer: failed to list *v1.ClusterIssuer: the server could not find the requested resource (get clusterissuers.cert-manager.io)

Although CluserIssuer is configured correctly and can issue certificates without any problem. The config map is:

apiVersion: v1
data:
  _example: |
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################

    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this block and unindented to actually change the configuration.

    # issuerRef is a reference to the issuer for this certificate.
    # IssuerRef should be either `ClusterIssuer` or `Issuer`.
    # Please refer `IssuerRef` in https://github.com/jetstack/cert-manager/blob/master/pkg/apis/certmanager/v1alpha1/types_certificate.go
    # for more details about IssuerRef configuration.
    issuerRef: |
      kind: ClusterIssuer
      name: letsencrypt-issuer
  issuerRef: |
    kind: ClusterIssuer
    name: letsencrypt-prod
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"_example":"################################\n#                              #\n#    EXAMPLE CONFIGURATION     #\n#                              #\n################################\n\n# This block is not actually functional configuration,\n# but serves to illustrate the available configuration\n# options and document them in a way that is accessible\n# to users that `kubectl edit` this config map.\n#\n# These sample configuration options may be copied out of\n# this block and unindented to actually change the configuration.\n\n# issuerRef is a reference to the issuer for this certificate.\n# IssuerRef should be either `ClusterIssuer` or `Issuer`.\n# Please refer `IssuerRef` in https://github.com/jetstack/cert-manager/blob/master/pkg/apis/certmanager/v1alpha1/types_certificate.go\n# for more details about IssuerRef configuration.\nissuerRef: |\n  kind: ClusterIssuer\n  name: letsencrypt-issuer\n","issuerRef":"kind: ClusterIssuer\nname: letsencrypt-prod\n"},"kind":"ConfigMap","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"65fab29b62204b780c436b3a223ce05b565b92f2"},"labels":{"kustomize.toolkit.fluxcd.io/name":"knative","kustomize.toolkit.fluxcd.io/namespace":"flux-system","networking.knative.dev/certificate-provider":"cert-manager","serving.knative.dev/release":"v0.25.1"},"name":"config-certmanager","namespace":"knative-serving"}}
    kustomize.toolkit.fluxcd.io/checksum: 65fab29b62204b780c436b3a223ce05b565b92f2
  creationTimestamp: "2021-09-15T09:31:33Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: knative
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    networking.knative.dev/certificate-provider: cert-manager
    serving.knative.dev/release: v0.25.1
  name: config-certmanager
  namespace: knative-serving
  resourceVersion: "12178"
  uid: 8baecfdc-940e-4565-898d-94e05ebb6160

ClusterIssuer config is:

apiVersion: v1
items:
- apiVersion: cert-manager.io/v1alpha2
  kind: ClusterIssuer
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1alpha2","kind":"ClusterIssuer","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"4a9fba3cb1dad95a8c7fdc4716a6662b03962839"},"labels":{"kustomize.toolkit.fluxcd.io/name":"certs","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"letsencrypt-prod"},"spec":{"acme":{"email":"caspiandb@users.noreply.github.com","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-v02.api.letsencrypt.org/directory","solvers":[{"dns01":{"route53":{"region":"us-east-1"}}}]}}}
      kustomize.toolkit.fluxcd.io/checksum: 4a9fba3cb1dad95a8c7fdc4716a6662b03962839
    creationTimestamp: "2021-09-15T09:31:02Z"
    generation: 1
    labels:
      kustomize.toolkit.fluxcd.io/name: certs
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: letsencrypt-prod
    resourceVersion: "11765"
    uid: 0e909fe7-1eed-4da7-9a31-65fd2c08d991
  spec:
    acme:
      email: caspiandb@users.noreply.github.com
      privateKeySecretRef:
        name: letsencrypt-prod
      server: https://acme-v02.api.letsencrypt.org/directory
      solvers:
      - dns01:
          route53:
            accessKeyID: ""
            region: us-east-1
            role: ""
            secretAccessKeySecretRef:
              name: ""
  status:
    acme:
      lastRegisteredEmail: caspiandb@users.noreply.github.com
      uri: https://acme-v02.api.letsencrypt.org/acme/acct/201519460
    conditions:
    - lastTransitionTime: "2021-09-15T09:31:03Z"
      message: The ACME account was registered with the ACME server
      reason: ACMEAccountRegistered
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Finally the networking config map:

apiVersion: v1
data:
  _example: |
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################

    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this example block and unindented to be in the data block
    # to actually change the configuration.

    # ingress.class specifies the default ingress class
    # to use when not dictated by Route annotation.
    #
    # If not specified, will use the Istio ingress.
    #
    # Note that changing the Ingress class of an existing Route
    # will result in undefined behavior.  Therefore it is best to only
    # update this value during the setup of Knative, to avoid getting
    # undefined behavior.
    ingress.class: "istio.ingress.networking.knative.dev"

    # certificate.class specifies the default Certificate class
    # to use when not dictated by Route annotation.
    #
    # If not specified, will use the Cert-Manager Certificate.
    #
    # Note that changing the Certificate class of an existing Route
    # will result in undefined behavior.  Therefore it is best to only
    # update this value during the setup of Knative, to avoid getting
    # undefined behavior.
    certificate.class: "cert-manager.certificate.networking.knative.dev"

    # domainTemplate specifies the golang text template string to use
    # when constructing the Knative service's DNS name. The default
    # value is "{{.Name}}.{{.Namespace}}.{{.Domain}}".
    #
    # Valid variables defined in the template include Name, Namespace, Domain,
    # Labels, and Annotations. Name will be the result of the tagTemplate
    # below, if a tag is specified for the route.
    #
    # Changing this value might be necessary when the extra levels in
    # the domain name generated is problematic for wildcard certificates
    # that only support a single level of domain name added to the
    # certificate's domain. In those cases you might consider using a value
    # of "{{.Name}}-{{.Namespace}}.{{.Domain}}", or removing the Namespace
    # entirely from the template. When choosing a new value be thoughtful
    # of the potential for conflicts - for example, when users choose to use
    # characters such as `-` in their service, or namespace, names.
    # {{.Annotations}} or {{.Labels}} can be used for any customization in the
    # go template if needed.
    # We strongly recommend keeping namespace part of the template to avoid
    # domain name clashes:
    # eg. '{{.Name}}-{{.Namespace}}.{{ index .Annotations "sub"}}.{{.Domain}}'
    # and you have an annotation {"sub":"foo"}, then the generated template
    # would be {Name}-{Namespace}.foo.{Domain}
    domainTemplate: "{{.Name}}.{{.Namespace}}.{{.Domain}}"

    # tagTemplate specifies the golang text template string to use
    # when constructing the DNS name for "tags" within the traffic blocks
    # of Routes and Configuration.  This is used in conjunction with the
    # domainTemplate above to determine the full URL for the tag.
    tagTemplate: "{{.Tag}}-{{.Name}}"

    # Controls whether TLS certificates are automatically provisioned and
    # installed in the Knative ingress to terminate external TLS connection.
    # 1. Enabled: enabling auto-TLS feature.
    # 2. Disabled: disabling auto-TLS feature.
    autoTLS: "Disabled"

    # Controls the behavior of the HTTP endpoint for the Knative ingress.
    # It requires autoTLS to be enabled.
    # 1. Enabled: The Knative ingress will be able to serve HTTP connection.
    # 2. Disabled: The Knative ingress will reject HTTP traffic.
    # 3. Redirected: The Knative ingress will send a 301 redirect for all
    # http connections, asking the clients to use HTTPS.
    httpProtocol: "Enabled"

    # rolloutDuration contains the minimal duration in seconds over which the
    # Configuration traffic targets are rolled out to the newest revision.
    rolloutDuration: "0"

    # autocreateClusterDomainClaims controls whether ClusterDomainClaims should
    # be automatically created (and deleted) as needed when DomainMappings are
    # reconciled.
    #
    # If this is "false" (the default), the cluster administrator is
    # responsible for creating ClusterDomainClaims and delegating them to
    # namespaces via their spec.Namespace field. This setting should be used in
    # multitenant environments which need to control which namespace can use a
    # particular domain name in a domain mapping.
    #
    # If this is "true", users are able to associate arbitrary names with their
    # services via the DomainMapping feature.
    autocreateClusterDomainClaims: "false"

    # If true, networking plugins can add additional information to deployed
    # applications to make their pods directly accessible via their IPs even if mesh is
    # enabled and thus direct-addressability is usually not possible.
    # Consumers like Knative Serving can use this setting to adjust their behavior
    # accordingly, i.e. to drop fallback solutions for non-pod-addressable systems.
    #
    # NOTE: This flag is in an alpha state and is mostly here to enable internal testing
    #       for now. Use with caution.
    enable-mesh-pod-addressability: "false"

    # Defines the scheme used for external URLs if autoTLS is not enabled.
    # This can be used for making Knative report all URLs as "HTTPS" for example, if you're
    # fronting Knative with an external loadbalancer that deals with TLS termination and
    # Knative doesn't know about that otherwise.
    defaultExternalScheme: "http"
  autoTLS: Enabled
  domainTemplate: '{{.Name}}-{{.Namespace}}.{{.Domain}}'
  httpProtocol: Redirected
  ingress.class: contour.ingress.networking.knative.dev
kind: ConfigMap
metadata:
  annotations:
    knative.dev/example-checksum: d38faef1
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"_example":"################################\n#                              #\n#    EXAMPLE CONFIGURATION     #\n#                              #\n################################\n\n# This block is not actually functional configuration,\n# but serves to illustrate the available configuration\n# options and document them in a way that is accessible\n# to users that `kubectl edit` this config map.\n#\n# These sample configuration options may be copied out of\n# this example block and unindented to be in the data block\n# to actually change the configuration.\n\n# ingress.class specifies the default ingress class\n# to use when not dictated by Route annotation.\n#\n# If not specified, will use the Istio ingress.\n#\n# Note that changing the Ingress class of an existing Route\n# will result in undefined behavior.  Therefore it is best to only\n# update this value during the setup of Knative, to avoid getting\n# undefined behavior.\ningress.class: \"istio.ingress.networking.knative.dev\"\n\n# certificate.class specifies the default Certificate class\n# to use when not dictated by Route annotation.\n#\n# If not specified, will use the Cert-Manager Certificate.\n#\n# Note that changing the Certificate class of an existing Route\n# will result in undefined behavior.  Therefore it is best to only\n# update this value during the setup of Knative, to avoid getting\n# undefined behavior.\ncertificate.class: \"cert-manager.certificate.networking.knative.dev\"\n\n# domainTemplate specifies the golang text template string to use\n# when constructing the Knative service's DNS name. The default\n# value is \"{{.Name}}.{{.Namespace}}.{{.Domain}}\".\n#\n# Valid variables defined in the template include Name, Namespace, Domain,\n# Labels, and Annotations. Name will be the result of the tagTemplate\n# below, if a tag is specified for the route.\n#\n# Changing this value might be necessary when the extra levels in\n# the domain name generated is problematic for wildcard certificates\n# that only support a single level of domain name added to the\n# certificate's domain. In those cases you might consider using a value\n# of \"{{.Name}}-{{.Namespace}}.{{.Domain}}\", or removing the Namespace\n# entirely from the template. When choosing a new value be thoughtful\n# of the potential for conflicts - for example, when users choose to use\n# characters such as `-` in their service, or namespace, names.\n# {{.Annotations}} or {{.Labels}} can be used for any customization in the\n# go template if needed.\n# We strongly recommend keeping namespace part of the template to avoid\n# domain name clashes:\n# eg. '{{.Name}}-{{.Namespace}}.{{ index .Annotations \"sub\"}}.{{.Domain}}'\n# and you have an annotation {\"sub\":\"foo\"}, then the generated template\n# would be {Name}-{Namespace}.foo.{Domain}\ndomainTemplate: \"{{.Name}}.{{.Namespace}}.{{.Domain}}\"\n\n# tagTemplate specifies the golang text template string to use\n# when constructing the DNS name for \"tags\" within the traffic blocks\n# of Routes and Configuration.  This is used in conjunction with the\n# domainTemplate above to determine the full URL for the tag.\ntagTemplate: \"{{.Tag}}-{{.Name}}\"\n\n# Controls whether TLS certificates are automatically provisioned and\n# installed in the Knative ingress to terminate external TLS connection.\n# 1. Enabled: enabling auto-TLS feature.\n# 2. Disabled: disabling auto-TLS feature.\nautoTLS: \"Disabled\"\n\n# Controls the behavior of the HTTP endpoint for the Knative ingress.\n# It requires autoTLS to be enabled.\n# 1. Enabled: The Knative ingress will be able to serve HTTP connection.\n# 2. Disabled: The Knative ingress will reject HTTP traffic.\n# 3. Redirected: The Knative ingress will send a 301 redirect for all\n# http connections, asking the clients to use HTTPS.\nhttpProtocol: \"Enabled\"\n\n# rolloutDuration contains the minimal duration in seconds over which the\n# Configuration traffic targets are rolled out to the newest revision.\nrolloutDuration: \"0\"\n\n# autocreateClusterDomainClaims controls whether ClusterDomainClaims should\n# be automatically created (and deleted) as needed when DomainMappings are\n# reconciled.\n#\n# If this is \"false\" (the default), the cluster administrator is\n# responsible for creating ClusterDomainClaims and delegating them to\n# namespaces via their spec.Namespace field. This setting should be used in\n# multitenant environments which need to control which namespace can use a\n# particular domain name in a domain mapping.\n#\n# If this is \"true\", users are able to associate arbitrary names with their\n# services via the DomainMapping feature.\nautocreateClusterDomainClaims: \"false\"\n\n# If true, networking plugins can add additional information to deployed\n# applications to make their pods directly accessible via their IPs even if mesh is\n# enabled and thus direct-addressability is usually not possible.\n# Consumers like Knative Serving can use this setting to adjust their behavior\n# accordingly, i.e. to drop fallback solutions for non-pod-addressable systems.\n#\n# NOTE: This flag is in an alpha state and is mostly here to enable internal testing\n#       for now. Use with caution.\nenable-mesh-pod-addressability: \"false\"\n\n# Defines the scheme used for external URLs if autoTLS is not enabled.\n# This can be used for making Knative report all URLs as \"HTTPS\" for example, if you're\n# fronting Knative with an external loadbalancer that deals with TLS termination and\n# Knative doesn't know about that otherwise.\ndefaultExternalScheme: \"http\"\n","autoTLS":"Enabled","domainTemplate":"{{.Name}}-{{.Namespace}}.{{.Domain}}","httpProtocol":"Redirected","ingress.class":"contour.ingress.networking.knative.dev"},"kind":"ConfigMap","metadata":{"annotations":{"knative.dev/example-checksum":"d38faef1","kustomize.toolkit.fluxcd.io/checksum":"65fab29b62204b780c436b3a223ce05b565b92f2"},"labels":{"kustomize.toolkit.fluxcd.io/name":"knative","kustomize.toolkit.fluxcd.io/namespace":"flux-system","serving.knative.dev/release":"v0.25.1"},"name":"config-network","namespace":"knative-serving"}}
    kustomize.toolkit.fluxcd.io/checksum: 65fab29b62204b780c436b3a223ce05b565b92f2
  creationTimestamp: "2021-09-15T09:31:33Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: knative
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    serving.knative.dev/release: v0.25.1
  name: config-network
  namespace: knative-serving
  resourceVersion: "12187"
  uid: 69a1e18a-37ef-4262-9408-b71abdcb8a41

knative-extensions / net-certmanager

cannot issue any auto certificates, although manual cert-manager certs work ok #293