Is there a way to reference the values of resources created with kcc?

[red8888]

This is specifically I'm working around: https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing

I want to create internal load balancers and attach a GCP reserved private IP address. This is frustratingly not possible (I don't know why gcp doesnt support this boooo), but what I can do is create a reserved private IP and assign the IP of that reserved IP to the load balancer via loadBalancerIP like this:

---
apiVersion: v1
kind: Service
metadata:
  name: myservice
  annotations:
    cloud.google.com/load-balancer-type: Internal
spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.123.123 # I can get this value from a reserved IP I previously created
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: app
    protocol: TCP

I can create a reserved private IP with kcc, but can I reference the value of that IP in the same manifest like this?

---
  apiVersion: compute.cnrm.cloud.google.com/v1beta1
  kind: ComputeAddress
  metadata:
    name: myip
  spec:
    addressType: INTERNAL
    location: us-west2
    ipVersion: IPV4
    subnetworkRef:
      external: https://www.googleapis.com/compute/v1/projects/myproject/regions/us-west2/subnetworks/mynetwork

---
apiVersion: v1
kind: Service
metadata:
  name: myservice
  annotations:
    cloud.google.com/load-balancer-type: Internal
spec:
  type: LoadBalancer
  # Get the value of the "Address" field from the myip resource
  loadBalancerIP: myip.Spec.Address
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: app
    protocol: TCP

Is something like this possible? I would be useful for many other resources as well if it is

[maqiuyujoyce]

Hi @red8888 , thank you for your question and sorry for the frustration you've been through. Unfortunately, references like myip.Spec.Address is not supported in Config Connector. Currently, an additional step is needed when you want to reference to a reserved IP address in other Kubernetes objects.

We've already received some other requests about exposing a generated IP address to a non Config Connector resource, e.g. issue #88 , and we are looking into potential options. Because the IP address is referenced outside of the scope of Config Connector, we'll need to use a different resource as the "bridge" for other Kubernetes resources to consume the IP address. One option would be that Config Connector creates a Kubernetes Service to expose the generated IP address. Does it sound like a working solution for you? (We might land on a different solution, but this is something we are considering.)

[red8888]

My specific issue with the resource is really that gcp doesnt support reserved IPs for internal LBs yet. If it did I could just add an annotation to the service like I can with the ingress to use a reserved IP.

But generally speaking maybe this is more a k8s limitation than anything else? Is referencing these sorts of properties between CRDs not supported natively?

Frankly I'm too ignorant of the internals to know what the right solution is. The nicest user experience would be what I explained in my question (referencing the value of a field of a resource that k8s hasn't created yet)- perhaps that is just not possible. This is a common patter of IaC tools I have used like terraform, cloudformation, etc so thats why I natively asked if it was possible but seems like its not? For example in terraform I can create a new VPC and in the same config reference the ID of that VPC (whatever it is when its created) for configuring other resources.

[maqiuyujoyce]

Hi @red8888 , thank you for your questions and suggestions.

Referencing fields among resources is not natively supported in K8s, so Config Connector needs to design and implement the support of those scenarios from scratch. For example, the resource reference in Config Connector is supported via the [field]Ref fields.

Referencing the generated IP is another common request we've received and we are looking into how to better support it. Because of the limitation of K8s, we cannot make a K8s Service reference to a Config Connector resource, and the UX will likely be different from other IaC tools you've used. That's why we are proposing that we can have Config Connector create a Kubernetes Service to expose the generated IP address. Feel free to share your comments with us so that we know if our proposal works for you.

[tonybenchsci]

@maqiuyujoyce Would it be more scaleable (for all current and future resource types) to adopt something more generic like IBM/composable kinds? Or do you have optimism that KCC will roll out some mechanism to expose all auto-generated spec/status fields that may be needed as inputs to other resources?

You can see how end-to-end IaC GitOps and automation will be made possible this way. Currently with KCC, pre-naming sources and using resourceRef solves half of the problem; the other half is being able to use resource[spec.field, status.field]Ref

[toumorokoshi]

Hey @tonybenchsci! overall we are looking at trying to find the right general abstractions to enable better composability.

Specifically with IP addresses, we're looking at potentially providing some intermediary object to expose the IP address to be referenced by others, such as Endpoints or EndpointSlices. Of course this would require the Service kind above to be able to consume an IP address by an Endpoint.

More generally we're investigating with partner tools such as kpt to expose some way to do arbitrary references and populating of values client-side.

Would it be more scaleable (for all current and future resource types) to adopt something more generic like IBM/composable kinds?

Do you have links or an example?

Or do you have optimism that KCC will roll out some mechanism to expose all auto-generated spec/status fields that may be needed as inputs to other resources?

I don't think we'd ever go as far as exposing every spec and status field as some referencable resource. So we're working with other Kubernetes folks to figure out if there's a better pattern for the more general references that we need to construct to provide a good IaC experience via Kubernetes.

[bshihr]

@toumorokoshi have "primitives" like ConfigMaps and or Secrets been considered as an intermediary for composability? This would seem natural fit for the application layer e.g. IAPIdentityAwareProxyClient status is usually consumed by injecting into a binary (via command-line args, env vars, etc). However, it doesn't address composing between primitives (your example of a Service derived from some kind of KCC CRD).

[toumorokoshi]

@toumorokoshi have "primitives" like ConfigMaps and or Secrets been considered as an intermediary for composability? This would seem natural fit for the application layer e.g. IAPIdentityAwareProxyClient status is usually consumed by injecting into a binary (via command-line args, env vars, etc). However, it doesn't address composing between primitives (your example of a Service derived from some kind of KCC CRD).

Definitely! I think for places where there are clear intermediary resources, we'll try to use them.

One area we're also thinking about is whether references need to be more generic: it may be difficult to play whack-a-mole by continuing to add these abstractions one by one, resource per resource.

update on priority: we haven't been able to prioritize looking into this work, but we know this is a pretty big pain point and will probably start sketching some designs to scope the work in the next quarter or two.

[mhubig]

I came here looking for a way to get some metadata (mostly the ip address) from a KCC created Cloud SQL database into my workload. I know I can workaround this by using the Google Cloud SQL Proxy, but a more general approach that works with all KCC resources would be really nice!

One possible way to get those kind of composability would be to combine ConfigMaps and Secrets with a way to specify outputs like it's done by terraform.

I would imagine an outputs API which could be used to instruct the KCC controller to create a ConfigMap containing the specified output variables. One could then use this config map to inject the output variables into the workloads. One additional benefit of this solution would be an implicit dependency between the KCC resource and the workloads.

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: mysql-private-instance
spec:
  databaseVersion: MYSQL_5_7
  region: us-central1
  settings:
    tier: db-f1-micro
    ipConfiguration:
      ipv4Enabled: true
      privateNetworkRef:
        name: mysql-private-network
  outputs:
    configmap: mysql-private-instance
    fields:
    - name: IP_ADDRESS
      value: this.ipv4_address

[raffomartini]

If you use kpt to package the manifest the apply-time-mutation feature has recently been rolled in to work around the limitation. If you don't, I highly recommend it, it works nicely with kcc.

[jcanseco]

Thank you @raffomartini! Yes kpt's apply-time-mutation is what we'd recommend customers to use to solve this problem. The apply-time-mutation feature was specifically designed to solve this problem (we work with the kpt team closely), and it was meant to work with other K8s tooling beyond just KCC.

We understand not all customers use kpt and some would find a more native solution more preferable (thanks for the design idea @mhubig!). However, we are unfortunately not there yet, so we recommend customers to try out kpt if they can. Some of the tool's features are meant to solve common orchestration problems that people face when using KCC, such as this one.

[RolandOtta]

we have a similar requirement as described in this issue.

we are creating a redis instance with kcc and would like also to create proper DNS records for the created instances.

so far we are using kpt to render the yaml files from a blueprint and are pushing those yaml files to a repository that is used by kcc to sync the resources.

how does the apply-time-mutation mentioned by @raffomartini come into play in such a scenario? i am confused because the docs mention that the apply-time-mutation is only enforced by the kpt live apply and kpt live destroy commands. i am not familiar with those commands, but are these somehow relevant in case you are working with kcc?

[caieo]

Hi @RolandOtta, we're sorry that we don't currently support your use case, but you should be able to accomplish your scenario with kpt the way @raffomartini mentioned. When a RedisInstance is created, the IP address (I assume you want to record) will appear in the status. Looking at the example from the kpt live apply documentation, you can utilize the kpt notation to indicate what the target field you want to populate is and just use that in the DNS record. That being said, we are not experts with using kpt, so it might be more helpful to ask about kpt live apply directly to the kpt team via their contact page or their GitHub.

[wleese]

Note that we found that Config Sync, doesn't not support kpt's apply-time-mutation at this point in time

[RolandOtta]

well it is technically available since 1.11.0+ since support for the depends-on annotation has been added.

the problem is, that the remediator reverts the mutations when it runs.

so you can use it in combination with the client.lifecycle.config.k8s.io/mutation: ignore annotation i guess. but it is a dirty workaround.

i opened a ticket a google again for supporting apply-time-mutation in acm ... hopefully this gets done

[Rahul1804]

KPT apply-time-muatation works only if require dependencies are in the same resource group. ComputeAddress and Service resource are not in same resource group. link - https://github.com/kubernetes-sigs/cli-utils/blob/ed4ec48b3405206c6c2200adbbc9606ea46b4257/pkg/object/graph/depends.go#LL242C7-L242C61

[diviner524]

@karlkfi Could you please help on the kpt apply-time-mutation question below? Is this a known limitation?

KPT apply-time-muatation works only if require dependencies are in the same resource group. ComputeAddress and Service resource are not in same resource group. link - https://github.com/kubernetes-sigs/cli-utils/blob/ed4ec48b3405206c6c2200adbbc9606ea46b4257/pkg/object/graph/depends.go#LL242C7-L242C61

[karlkfi]

You've misunderstood the constraint.

Objects do not need to be in the same API resource group. They need to be in the same ResourceGroup, which is the resource kpt uses to track inventory of applied objects. That means the two objects need to be applied at the same time with kpt, from the same directory tree.