concat function

[philippemerle]

The TOSCA specification says that The concat function is used to concatenate two or more string values within a TOSCA service template.

For the following service template

tosca_definitions_version: tosca_simple_yaml_1_3
node_types:
  MySoftware:
    derived_from: tosca.nodes.SoftwareComponent
    capabilities:
      endpoint:
        type: tosca.capabilities.Endpoint
topology_template:
  node_templates:
    compute:
      type: tosca.nodes.Compute
      attributes:
        public_address: my.compute.com
    software:
      type: MySoftware
      capabilities:
        endpoint:
          properties:
            protocol: http
            port: 3000
            url_path: api/v1/software
      requirements:
        - host: compute
  outputs:
    my_software_endpoint_url:
      type: string
      value:
        concat:
          - get_property: [software, endpoint, protocol]
          - "://"
          - get_attribute: [compute, public_address]
          - ":"
          - get_property: [software, endpoint, port]
          - "/"
          - get_property: [software, endpoint, url_path]

one could expect that the value of the my_software_endpoint_url output will be http://my.compute.com:3000/api/v1/software.

But here, the 5th parameter of the concat function is not a string but a PortDef integer.

Processing this service template with different TOSCA parsers produces different behaviors:

• No error with Puccini
• No error with Ubicity Validator after removing the line type: string else one parsing error
• No error with xOpera parser but one runtime exception when trying to get the outputs after deploying the template
• One typing error (i.e. the 5th parameter is not a string) with Cloudnet TOSCA toolbox

So the given service template is not portable to various TOSCA implementations.

Then the specification of the TOSCA concat function should be improved to clearly define that:

• each parameter SHALL BE of type string XOR each parameter SHALL BE convertible to a string, e.g. a PortDef integer could be stringified
• if convertible, which types can be convertible to a string? Is a complex data type convertible?
• what happens if a parameter refers to an unset attribute/property? Is it a parsing error? is it a runtime error? Does the parameter set to an empty string? other behavior?

Any opinions?

[lauwers]

The Ubicity orchestrator incorrectly flagged the “type” keyword in the output definition as an error. We have fixed this in the latest version.

You are correct that the spec needs to clearly define whether types other than strings can be used in concat (and other) functions.

[philippemerle]

You are correct that the spec needs to clearly define whether types other than strings can be used in concat (and other) functions.

I propose that at least YAML Types (string, integer, float, boolean, timestamp), TOSCA version, and TOSCA scalar-unit types could be used in concat functions. Other types (null, range, list, map, complex data type) could be also used but it will be needed to define precisely how they are stringified.

[tliron]

As I stated elsewhere, I think TOSCA should treat functions like dynamic languages (Python, Ruby, JavaScript) do. That is: arguments can be of any type and return values can be of any type. This allows functions to have their own special semantics. For example, a function can return one type in one circumstance and another type in another circumstance. get_property indeed works that way.

Still, we do need to make sure that internal functions are properly documented. I agree with @philippemerle that allowing the stringification of primitive types is a good idea, but even that has to be documented. For example, there is no one way to stringify floats or even large integers (exponential format). Complex type stringification is too complex, I propose not supporting that.

[tliron]

On second thought, I would like to make another point -- I don't think TOSCA parsers should check the return type or value at all during design-time validation at all, even for a "static" function like get_property (indeed, as you noticed Puccini does not normally test for this, unless you specifically ask to --coerce functions). The reason is that even if the semantics of a function (like built-in functions) is well known, at best we can test for the general type. But because we might not have the value itself until runtime (get_attribute) then we can't check if the value adheres to other constraints (e.g. max_length). So, I would say that whatever value you get from such design-time validation is quite limited and provides not a lot of confidence.

And the implementation could be complex -- because functions can be nested, you would have to mark any expression that has get_attribute at any level of its nesting as "can't validate its value". If we add support for custom functions, too, we would have to add some way to mark them as either "dynamic" (like get_attribute) or "static" (like concat) for this purpose. I don't think this added complexity is worth the limited benefits.

My design principle with Puccini is that anytime a user uses a function it essentially cannot be validated until deployment time, even for seemingly "static" functions. Yes, I know I advocate strongly for putting as much design-time validation as possible in TOSCA, but even I try to choose my battles carefully. :)

[lauwers]

Ubicity checks whether the return types of a function is compatible with the type of the property for which it is defined as the value. It handles nested functions, and it also flags cases where a function value for a mandatory property references an optional value.

[tliron]

That's great, but the question is whether we want to standardize this behavior. The cost can be very significant. For Puccini it would mean that for every function (including custom functions) there would need be both the regular function, as well as a special "validation function" path that calculates the return type. As I stated before, the usefulness of this return type validation is rather limited -- without an actual value we can't validate whether it would adhere to constraints on that type.

Having to add this companion function to every function for the rather limited usefulness of having a strictly primitive return type seems like far too much of a cost. Even if this were standardized in TOSCA I would not implement it. To be clear, the errors would appear in Puccini -- but only when functions are explicitly called, which happens during Day 1 and 2, not during Day 0 (unless specifically enforced with the --coerce argument, which of course cannot handle get_attribute calls very meaningfully beyond their default values).

[lauwers]

I’m not suggesting we should standardize this. However, when we revisit the discussion about custom functions, we should discuss whether these custom functions should define their return type (my vote would likely be yes).

[tliron]

My vote would be no because I would like custom functions to have the same heuristic freedom as the built-in functions, e.g. get_attribute having a different return type depending on the attribute. I can see custom functions as a way to plug in custom graph traversal languages, not just for the TOSCA topology, but also for various embedded or external data sources.

As stated before, I would like TOSCA function to work like functions in dynamic languages (Python, Ruby, JavaScript, etc.). The function has a name. The function has a variable number of arguments. The function returns a single value, which can be of any type. This is a very familiar calling mechanism that would be easily understood by users.

[philippemerle]

Ubicity checks whether the return types of a function is compatible with the type of the property for which it is defined as the value. It handles nested functions, and it also flags cases where a function value for a mandatory property references an optional value.

Cloudnet TOSCA toolbox does the same checks.