Proposal: Allow variable Function inputs

[bobh66]

What problem are you facing?

Composition Functions can receive input from the following sources: (thanks @negz )

• The function inputs, read from the Composition (hard-coded)
• The XR spec/status (both observed and pending mutations from other functions)
• Composed resource spec/status (both observed and pending mutations from other functions)
• Context passed from other functions

This means that in order for functions to handle variable inputs, they must do at least one of the following:

• be aware of the XR structure so they can retrieve observed/desired data
• accept field paths as inputs and retrieve the specified fields from the XR/composed data in the function
• be preceded by a function that retrieves the data and places it in a known location in the context so the function can retrieve it from that location

None of these options are particularly desirable for generic functions that want to be:

• XR-independent for maximum reuse
• simple to implement
• able to run standalone without dependencies on other functions

How could Crossplane help solve your problem?

Crossplane could support the ability to map fields from the XR to inputs of a Function and do the required value retrieval before calling the function.

For example, if a function accepts three inputs:

input:
  foo: 1
  bar: 2
  baz: 3

it would be very useful to be able to retrieve those values from the XR, for example:

inputRefs:
  foo: spec.parameters.foo
  bar: spec.parameters.bar

input:
  baz: 3

That would provide a lot more flexibility in the function pipeline without requiring functions to implement the value resolution logic. The functions shouldn't care where the input values come from.

[clementblaise]

I agree, and this would help achieve greater consistency between function inputs. Currently, functions have supported this in different ways (Field vs FromFieldPath):

• function-shell has shellCommandField
• function-extra-resources use valueFromFieldPath

While referencing from the XR adds flexibility, supporting input from the context would also be helpful, as it avoids passing data from the context to the XR. It would enable natively reading from EnrionmentConfig or extra resources. Is this something you imagine being supported @bobh66 ?

[bobh66]

While referencing from the XR adds flexibility, supporting input from the context would also be helpful, as it avoids passing data from the context to the XR. It would enable natively reading from EnrionmentConfig or extra resources. Is this something you imagine being supported @bobh66 ?

@clementblaise do you mean providing a means for passing context data into a Function that is not context-aware? For example function-environment-configs populates a specific context key apiextensions.crossplane.io/environment with the selected EnvironmentConfigs which is then available to all subsequent functions in the pipeline. However if the function is not context-aware then there is a need to be able to "patch" the information into the inputs from the context in the pipeline definition, for example:

inputRefs:
  foo: context["apiextensions.crossplane.io/environment"].foo
  bar: context["apiextensions.crossplane.io/environment"].bar

input:
  baz: 3

I think it's definitely reasonable to consider that case as part of this scenario.

[clementblaise]

@bobh66 Yes, right. I meant for non-context-aware functions. It's preferable as it avoids maintaining it on the functions side and allows the use of custom context keys. The example looks good.

[bobh66]

@negz I had an idea about how to implement this while avoiding more patch-like syntax. We can use the context as a fallback location for function inputs, keyed by the function name or the function's input CRD name, or something similar. If the pipeline step includes an input object for the function, that gets used when calling the function. If the pipeline step does not contain an input object, we can look for one in the context and use that if it is available.

That enables the composition developer to use something like function-go-templating (with https://github.com/crossplane-contrib/function-go-templating/issues/66 resolved) to write a specific key into the context using the CRD format of the input for a subsequent step in the pipeline, where we can invoke function-cel-filter with no inputs:

    pipeline:
    - functionRef:
        name: function-go-templating
      input:
        apiVersion: gotemplating.fn.crossplane.io/v1beta1
        inline:
          template: |-
            apiVersion: meta.gotemplating.fn.crossplane.io/v1alpha1
            kind: Context
            data:
              function-cel-filter:
                input:
                   filters:
                    - name: foo
                       expression: observed.composite.resource.spec.export == "bar"
    - functionRef:
        name: function-cel-filter
        # No input section is provided so Crossplane can look in the context to find a matching input type for this function name and use it when calling the function

I agree this feels a little bit like magic, but I think magic is bad when things happen without your knowledge and without having done anything to cause it. In this case you had to (a) populate the context in a specific format and (b) call the subsequent function without the input block. If you provide the input block then the context data is never referenced. If you don't populate the context AND don't provide the input then it is no different than what happens today.

One benefit to this approach is that the functions don't need to be modified to benefit from it - all of the "magic" happens in core Crossplane.

There are still some questions to be answered:

• what key should be used to ensure uniqueness?
• are there any concerns around having the same function called multiple times in the pipeline, possibly with a mixture of inputs specified and no input specified?

@clementblaise any thoughts?

[clementblaise]

@bobh66 I liked the simplicity of the initial implementation. If the goal is to avoid patch-like syntax, it would be acceptable to restrict the usage to only passing data. If data transformation is needed, it could be done upstream in the pipeline with the writing feature of function-go-templating, as you mentioned. Also, P&T could be updated to support writing in arbitrary contexts to facilitate transformations such as Convert, Map, Match, etc.

[bobh66]

@negz put together function-python which makes me wonder if it's possible to have a small python snippet that dynamically configures the inputs to another composition function and then makes the GRPC call to it, returning the results from the second function from the call to function-python. The advantage to this is that it doesn't require any changes to core Crossplane, the disadvantage is the additional layer of indirection and the fact that we don't know how hard it would be to establish mTLS to the second function from function-python. I'm hoping to try this if I have time, but time is always the problem.