How to expose information about created resources to other workloads in the cluster ?

[primeroz]

In my case i created a Redis Instance and would like to make the ip address automatically available to other workloads in the cluster so that i can actually use the redis instance itself.

• some resources create a Secret once they are reconciled, like the iam service account key one
• some other have predictable names like sql or buckets

But in the case of redis for example this is not possible and ip:port need to be made available to the workloads that will consume the redis instance ... ideally without having to wrap them in something that will fetch this information from kubernetes api server.

I was expecting / hoping that once the resource was reconciled i would get a configmap with the same fields as the resource status so that i could then use them in the workloads as ENV / volume mounts

kubectl get redisinstances.redis.cnrm.cloud.google.com -n sandbox redis-test1 -o yaml | yq read - 'status'
conditions:
- lastTransitionTime: "2020-02-18T16:06:26Z"
  message: The resource is up to date
  reason: UpToDate
  status: "True"
  type: Ready
createTime: "2020-02-18T16:02:11.444222567Z"
currentLocationId: europe-west1-d
host: 10.8.217.52
port: 6379

[TMaYaD]

Alternatively an admission controller that catches cnrm references in resources spec and replaces them would also work.

[TMaYaD]

@primeroz I found https://github.com/IBM/composable#namespaces which can be used to reference KCC resources.

[primeroz]

@TMaYaD interesting, your google-fu must be better then mine!

I will give it a shot, but unfortunately in my case this might not be a good solution because i push my manifests using flux gitops operator ... and so i kind of rely on the manifests to be created as their kind for flux to be able to properly track them and delete them

It might work though so i will surely try it

I actually had started to write a simple wrapper to run the entrypoint of a workload and be able to fetch any REF from any resource and export as env variable ...

to be honest though i really believe that an operator like the config connector should provide a solution to this problem out of the box to be really usable

[maqiuyujoyce]

Hi @primeroz, thank you for your feedback. We've also received the same request from another customer. We are currently looking into how to best handle this scenario. Will let you know when we have any updates.

[zarbis]

Any progress on this? Or at least some sort of workaround for a time being?

[mikebryant]

A use-case I have is getting the connection details for an SQLInstance out of the resource

[TMaYaD]

@mikebryant The way I handle it is by using composable with external service.

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: instance-name
spec:
  databaseVersion: POSTGRES_12
  region: us-central1
  settings:
    tier: db-custom-1-3840
    availabilityType: ZONAL
    ipConfiguration:
      ipv4Enabled: false
      privateNetworkRef:
        name: default
      requireSsl: false
    locationPreference:
      zone: us-central1-a
    backupConfiguration:
      enabled: true
---
# Turn into composable to skip kustomize from adding selectors
apiVersion: ibmcloud.ibm.com/v1alpha1
kind: Composable
metadata:
  name: cloudsql-instance-service
spec:
  template:
    kind: Service
    apiVersion: v1
    metadata:
      name: cloudsql-instance
    spec:
      type: ClusterIP
      ports:
      - port: 5432
        targetPort: 5432
---
apiVersion: ibmcloud.ibm.com/v1alpha1
kind: Composable
metadata:
  name: cloudsql-instance-endpoint
spec:
  template:
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: cloudsql-instance
    subsets:
    - addresses:
        - ip:
            getValueFrom:
              kind: SQLInstance
              name: instance-name
              path: '{.status.firstIpAddress}'
              format-transformers:
                - ToString
      ports:
        - port: 5432

[tonybenchsci]

@maqiuyujoyce Our team actually has this need for DNSRecordSets where we'd like to populate A-records in rrdatas fields by getting the generated outputs from ComputeAddress resources. Another use case the auto-generated writer-identity for GCP log-sinks, which we then need to grant writer permissions in a BigQuery datasets for analysis; note that this is actually possible (if in same project) using Deployment-Manager with

access:
    - role: roles/bigquery.dataOwner
      iamMember: $(ref.stackdriver-sink.writerIdentity)

[toumorokoshi]

Thanks for the use case! We'll keep that in mind when figuring out a solution.

[sf-vorlov]

Any update on this issue?

[gnagel]

Hi All, Any chance this one is moving forward soon? Ty

[diviner524]

@sf-vorlov and @gnagel, We have added some features in the DNSRecordSet CRD a while back, and it now supports one of the scenarios discussed above. See this link to a sample.

The more general feature request IIUC, is on how to extract information from one resource, and use it in another resource.

At this moment, our recommendation is to use the apply-time-mutation feature inkpt.

See related discussion here:

https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/334#issuecomment-1016384851

We are also actively exploring additional long-term solutions, such as implementing a native orchestration/composite layer in Config Connector. We will provide further updates as soon as they become available.

[sf-vorlov]

@diviner524 thanks for your update. We've already started using the rrdatasRefs/ComputeAddress feature - where the case calls for it. Otherwise we use the composable-operator, which allowed us to stay within the conventional helm charts path. Would be really nice if the composite feature is included.

[OperationalFallacy]

Hello,

I've been tracking the developments in this discussion since we're working on the GCP platform at an enterprise for more than a year. We're utilizing CDK8s to create resources with Config Connector, which significantly improved the developer's experience. However, the inability to reference output crashed it back. We often had to resort to manual hacks, double-stage deployments, or other workarounds.

I think it's time to address the elephant in the room. It's clear that if Config Connector intends to be a standard tool within GCP, a more holistic approach to this issue is necessary.

The focus should be on referencing resource outputs within whole application stacks with consistent ways to access these outputs across stacks. Infrastructure is an application. Having application set and access outputs is a fundamental requirement.

AWS CloudFormation solves this problem elegantly, particularly its Cloudformation "Outputs" which allow for the streamlined integration of resources across different stacks. I can put anything into the outputs, dynamic strings, URLs, endpoints, resources IDs and can read it from any other stack or app. Moreover, AWS CDK further builds upon this foundation, it offers programmatic solutions to reference resources between stacks, even in different regions.

For reference, AWS CloudFormation documentation: AWS CloudFormation Outputs Section.

[Coder63]

Hi , Do we have KCC fix for this limitation ? I don't want to use kpt & composable ..

[maqiuyujoyce]

@yuwenma fyi another frequently asked feature.

@barney-s do you think this can be a reasonable use case for composition?