Application syntax 2.0

[eeberhard]

Scope

This issue is to propose the next version of the AICA application syntax (version 2.0).

The goal is to define all the desired changes to the application syntax, including any breaking changes, to enable even better composition of applications. All proposals are up for discussion under this issue, including any syntax choices made in the examples.

Many of the suggested schema revisions would also require changes to modulo components, controllers, the Dynamic State Engine, and the implementation of the Developer Interface. This proposal does not go into details, but consider that some changes have more associated consequences in implementation work than others.

Outline of proposals

This proposal is structured in a certain way to group related concepts. This forms a narrative outline:

• A holistic approach to lifecycle states
• Automatic behaviors (auto-events)
• Restrict names to lower_snake_case
• Positions
• Display names
• Application events
• Static frames
• Component frames
• Metadata

Types of changes

The proposal could also be structured by the type of change. I have used the following labels throughout the proposal:

⚠️ BREAKING CHANGE ⚠️

Some parts of the proposal constitute breaking changes in the syntactical sense (old applications will not validate against the new schema).

🤔 CONCEPTUAL CHANGE 🤔

Some changes are conceptual changes even if not technically breaking (old applications may still validate against the new schema, but may not behave as expected).

💡 ENHANCEMENT 💡

Some changes are just ideas for enhancements that could be an eventual revision or addition to the schema.

Proposals

A holistic approach to lifecycle states

A main feature of an AICA application is that the states of all components and controllers can change dynamically (ergo, the name Dynamic State Engine). Since we already track the states (loaded or unloaded, including lifecycle states) of all elements, we could make better use of these states in conditional logic, and use the transitions of states to drive events.

State transitions as event drivers

Currently, we broadcast component lifecycle states through predicates (is_unconfigured, is_inactive, etc). We also broadcast a fake on_load predicate with every loaded component that is set true when the component is constructed, and automatically set false thereafter.

Component predicates in turn can be used to drive events. This is used commonly, especially in lifecycle components, either to automatically configure and activate themselves, or to drive certain behavior once active (see also automatic behaviors)

my_component:
  ...
  events:
    on_load:
      load: foo
    is_unconfigured:
      lifecycle: configure
    is_inactive:
      lifecycle: activate
    is_active:
      # do something else

There are multiple areas that could be improved:

• Components publish four lifecycle predicates on every step (inefficient)
• Confusion between "built-in" predicates and custom predicates (both in component descriptions and in the visualization of component nodes)
• Fake predicate on_load is published by the DSE, not the real component
• Fake predicate on_unload is consumed by the DSE but never broadcast (and only ever true after a component is unloaded, meaning it's not really part of the component state)
• Predicate logic is currently restricted to components; controller predicates are coming soon, but hardware interfaces themselves are unlikely to ever publish predicates (not on the ROS interface)

A bigger issue is that lifecycle predicates are used to indicate a state that is not necessarily just "on" and "off" but part of a more complex state diagram. Put plainly, the is_inactive predicate activates when the component is configured (from unconfigured state into the inactive state) but also when the component is deactivated (from active state into the inactive state). For application logic, the context of these transitions often matters; if the component is activated every time it is inactive, then it can never be deactivated without automatically re-activating itself.

The proposal to refactor this section is to remove lifecycle predicates and the fake on_[un]load predicates and instead use state transition triggers. The syntax might look something like:

⚠️ BREAKING CHANGE ⚠️ The events field would be restricted to specific, pre-defined state transitions

🤔 CONCEPTUAL CHANGE 🤔 A new predicates field would be responsible for predicate-driven events

my_component:
  events:
    on_load: ...
    on_configure: ...
    on_activate: ...
    on_deactivate: ...
    on_error: ...
  predicates:
    is_in_range: ...
    is_buffer_full: ...

The benefits I see are:

• State transition events are "directional", so arriving into the same state (e.g. inactive) triggers either the on_activate or on_deactivate events depending on context, but never both
• Syntax for state events more closely matches the actual API of the component implementations with state transition callbacks
• Syntax for state transition triggers more clearly distinguished from predicate names (generally a callback on_<transition> instead of a state is_<state>)
• "normal" state transition events are grouped (which are common to many components), while predicate driven events (which are specialized by each component) are grouped separately
• Tracking state can be applied to more than just components; the state transitions of controllers and hardware interfaces could equally be used to trigger events.

States in conditions

If we already track component and controller states, we can use them in conditions in much the same way we currently use predicates. Again, the main benefit is the distinction between common states across all components and custom predicates that are specific to some components.

🤔 CONCEPTUAL CHANGE 🤔 Condition objects would allow more variants including selection of specific states on various application components

conditions:
  my_condition:
    component: foo
    state: inactive
    ...
  my_condition:
    controller: bar
    hardware: baz
    state: active
    ...

States in sequence conditions

Similarly, we can use states in sequences either to wait for or assert a particular state.

🤔 CONCEPTUAL CHANGE 🤔 Sequence steps would allow more variants including selection of specific states on various application components

sequence:
  - wait:
      component: foo
      state: inactive
  - assert:
      controller: bar
      hardware: baz
      state: active

Automatic behaviors

In the developer interface, we have introduced a few concepts to reduce the number of event edge connections in the graph.

These are mainly auto-configure and auto-activate for lifecycle components, auto-load for controllers and auto-load for hardware.

For hardware, auto-load is a just a shortcut for on_start: {load: {hardware: ...}}, but for controllers and components, auto-events are driven by runtime behavior.

Auto-load for a controller means to load the controller right after the associated hardware interface is loaded. Auto-configure and auto-activate for a component respectively mean to trigger the lifecycle transition events from the unconfigured and inactive states.

🤔 CONCEPTUAL CHANGE 🤔 The common pattern of explicit auto-lifecycle triggers would be replaced by explicit fields

So instead of this:

my_component:
  ...
  events:
    is_unconfigured:
      lifecycle: configure
    is_inactive:
      lifecycle: activate

It could just be:

my_component:
  ...
  auto_configure: true
  auto_activate: true

For a controller, it would then be:

my_controller:
  auto_load: true

Is on_start still desirable?

We could entertain the idea of removing the on_start field entirely, and just defining that behavior in hardware, controllers and components directly

⚠️ BREAKING CHANGE ⚠️ If all application components declare their "auto-start" or "auto-load" status, the on_start field would be removed

components:
  my_component: ...
    component: ...
    auto_load: true

However, this is not a great choice in my opinion, because it makes the entry point to an application less obvious. I think even if "auto load on start" is represented differently in the visual graph, the information for loading on start should be contained in the application on_start block, not with each component individually.

Automatic looping of sequences

Sequences currently execute their steps only once. Once the sequence ends, it must be started again. This can be done either by a separate event caller or by using sequence: {restart: my_sequence} as the last sequence step.

sequences:
  my_sequence:
    - load: ...
    - unload: ...
    - sequence:
        restart: my_sequence

We can consider adding an explicit property to sequences to determine if they should loop by default. This would prevent the "loopback" event that the self-restart step would imply, just as in auto-lifecycle events.

⚠️ BREAKING CHANGE ⚠️ A sequence is currently defined as an array of steps. Adding any top-level fields changes this into an object and requires steps as a sub-property

sequences:
  my_sequence:
    loop: true
    steps:
      - load: ...

Restrict all names and namespaces to lower_snake_case

⚠️ BREAKING CHANGE ⚠️ Forbid upper case letters in any keywords (components, controllers, parameters, predicates, conditions, etc...)

Issue captured here:

• https://github.com/aica-technology/api/issues/140

Positional data in a separate field

I think it would be good to separate application logic (components, parameters, signal and event connections, ...) from visual properties like position.

In general, storing position data with the declaration of the component makes it less portable between applications (the absolute position is only meaningful relative to the rest of the application), and adds clutter. We might instead store position data separately under some positions field (exact syntax not formally defined):

⚠️ BREAKING CHANGE ⚠️ Definition of position: in components, hardware, etc would be forbidden

components:
  my_component:
    component: foo
    parameters: ...

positions:
  components:
    my_component:
      x: 100
      y: 200
  hardware:

More compact positional data

We might also consider that a dedicated position field could define the x, y coordinate more compactly as an array with two elements, rather than a map with x and y fields.

⚠️ BREAKING CHANGE ⚠️ Definition of x:, y: in positions would be forbidden

positions:
  components:
    my_component: [ 100, 200 ]

However I don't think this is a meaningful improvement, because re-generation of the YAML might still render it as a vertical list:

positions:
  components:
    my_component:
      - 100
      - 200

This takes up the same amount of vertical space but is less clear.

Positions for conditions and sequences

Conditions and sequences don't currently have position data. Again, these could be added to a new positions: field.

💡 ENHANCEMENT 💡 Position data for conditions and sequences could be added under a separate positions field

positions:
  conditions:
    my_condition:
      x: 100
      y: 200
  sequences:
    my_sequence:
      x: 100
      y: 200

However, if position data is to be associated with a sequence directly, it becomes a breaking change.

⚠️ BREAKING CHANGE ⚠️ A sequence is currently defined as an array of steps. Adding any top-level fields changes this into an object and requires steps as a sub-property

sequences:
  my_sequence:
    position:
      x: 100
      y: 200
    steps:
      - load: ...

Positional data for edges

A common pain is that the event and signal edges in the application graph become increasingly interconnected and difficult to render clearly in any deterministic and procedural way for complex applications. It would be desirable to be able to drag edges around and rearrange their routes to clean up the rendering of any application, similar to how nodes can be positioned manually. However, even if this feature is implemented, the edited edge positions should be persisted in the graph editor the next time that application is loaded, which implies that the visual routing information for the edges should be stored somewhere in the application.

The X,Y positions of components or hardware interfaces are currently stored next to the respective declaration of those nodes. Using a similar approach for edges is not a good idea for two reasons:

• Edges mainly connect two nodes; would the edge routing data be stored under the source or target node?
• The position of a node on the graph is a simple X, Y structure, while the routing information for an edge might be more complex (for example, an array of via points)

💡 ENHANCEMENT 💡 Edge information doesn't currently exist. If added to the schema in a new location, it would be a pure enhancement

positions:
  edges:
    react-flow-edge-id-1: [...]

Display names

Display names are also grouped with components and controllers at the moment. Like positions, there is an option to move them to a separate "visual" group, since they don't affect runtime behavior. Also, display names don't currently exist for conditions or sequences.

⚠️ BREAKING CHANGE ⚠️ Definition of display_name: in components, hardware, etc would be forbidden

components:
  my_component:
    component: foo
    parameters: ...

positions:
  components:
    my_component:
      x: 100
      y: 200
  hardware:

However, I don't think this idea can be justified as strongly as the case for positions, because the display name may remain portable with the component even if it is taken out of context of the application.

Display names for conditions and sequences

Conditions and sequences don't currently have a display_name. These could be added to a new display_names: field.

💡 ENHANCEMENT 💡 Display name data for conditions and sequences could be added under a separate field

display_names:
  conditions:
    my_condition: Critically cool condition
  sequences:
    my_sequence: Super snazzy sequence

However, if a display name is to be associated with a sequence directly, it becomes a breaking change.

⚠️ BREAKING CHANGE ⚠️ A sequence is currently defined as an array of steps. Adding any top-level fields changes this into an object and requires steps as a sub-property

sequences:
  my_sequence:
    display_name: Super snazzy sequence
    steps:
      - load: ...

Application events

There are currently no state events to control the running applications. It should be possible to stop the application from any event trigger within the application:

💡 ENHANCEMENT 💡

my_condition:
  events:
    application: stop

With the introduction of the application event keyword, we can then also consider other events (like pause, transition, etc)

Static frames

A significant part of robotic applications deals with frames (a.k.a. transforms, TF). We already have components that can record frames into a specific YAML format (frame recorder) and can broadcast them again (static frame broadcaster). What if this functionality was built in to the application?

💡 ENHANCEMENT 💡

frames:
  my_frame:
    orientation:
      w: 1.0
      x: 0.0
      y: 0.0
      z: 0.0
    position:
      x: 0.5
      y: 0.5
      z: 0.5
    reference_frame: world
  my_other_frame:
    orientation:
      w: 0.707
      x: 0.0
      y: 0.0
      z: 0.707
    position:
      x: 0.1
      y: 0.0
      z: 0.0
    reference_frame: my_frame

Defining these frames in the application would allow the frontend UI to visualize them, and it would also make it easier to associate frames with components through dropdown selection (see next section). One could also consider that an interactive UI would allow frames to be edited visually and update the frames saved in the application accordingly.

Components as consumers and producers of frames

Frame consumers

Often, a component consumes TF frames, as in the case of the Point Attractor. These are frequently provided as parameters:

my_component:
  ...
  parameters:
    center_frame: foo
    target_frame: bar
    reference_frame: baz # 

Instead, we could define a new frames field for components (and controllers) that defines named frames and reference frames for the component to use.

my_component:
  ...
  frames:
    center: foo  # by default, reference frame is world
    target:      # a frame could also be defined with a specific reference frame
      frame: bar
      reference_frame: baz

my_component:
  ...
  reference_frame: baz
  frames:
    center:
      frame: foo
      reference_frame: world
    target: bar

Just as with signals, the definition frames would ultimately just be a wrapper for internal parameters (so the center frame might be wrapped under a parameter center_frame), but would allow more abstraction for the UI integration.

For example, frames could be selected from a dropdown based on the globally available frames, or from a database of static frames.

It would also reduce complexity in using frames in components with current inconsistencies in declaring separate string parameters for frames and reference frames.

Frame producers

Some components broadcast frames dynamically. This is similar to outputs, but is harder to keep track of in the application.

Unlike consumers of frames, producers are free to choose whatever names they want for the frame. But, practically speaking, parameters are often used to assign the name of the output frame explicitly, especially in the case of multiple components.

We could consider defining these in a similar way in the schema:

my_component:
  ...
  output_frames:
    orbit: my_component_output_frame

With this consideration in mind, there could be three fields to define the default reference frame, the input frames, and the output frames respectively:

🤔 CONCEPTUAL CHANGE 🤔 This suggestion would not stop the old parameter approach from working for frames, but usage patterns underlying conventions in refactored components might change

my_component:
  ...
  reference_frame: baz
  input_frames:
    center:
      frame: foo
      reference_frame: world
    target: bar
  output_frames:
    orbit: my_component_output_frame

Extending component descriptions with frame declarations

💡 ENHANCEMENT 💡

To accompany this proposal, component and controller description schemas should also be extended to declare and describe the frames they consume and produce.

Metadata

💡 ENHANCEMENT 💡 Metadata is not currently included in the application schema, so all of these points are simple enhancements

Syntax version

Applications are authored with a particular application syntax version, which in turn determines which version of the AICA framework it is compatible with. As the application syntax is updated, some old applications may become outdated (for example, this proposal is about making major changes to the syntax to create a version 2-0-0). The syntax version of outdated applications might not be obvious to determine automatically unless analyzed closely. Between application syntax versions, it may be possible to perform automatic migration.

The application syntax version can already be captured with a meta comment in the yaml document which is used by the VSCode yaml extension:

# yaml-language-server: $schema=https://docs.aica.tech/schemas/1-2-0/application.schema.json

However, this format for syntax version is not explicitly provided by the application schema, and can easily be stripped or skipped by parsers because it acts as a comment.

A metadata property could therefore include the syntax version of the application schema used:

schema: 2-0-0

Package dependencies

An application might rely on specific component or hardware packages. Capturing these dependencies and their versions could help automatically detect compatibility issues.

Some tentative syntax variants:

packages:
  - "@aica/components/auto-frame-calibration = v1.0.0"
  - "@aica/collections/ur-collection = v3.1.0"

packages:
  - {package: "@aica/components/auto-frame-calibration", version: "v1.0.0"}
  - {package: "@aica/collections/ur-collection", version: "v3.1.0"}

Name, description, author...

Other descriptive metadata may be useful to add to the schema, although it wouldn't have a functional role (in contrast to syntax or dependency declarations)

metadata:
  name: My application
  description: Some description of the application
  author: Example FooBarhard

[domire8]

Exciting stuff and a lot of great ideas! Here's my take:

A holistic approach to lifecycle states

• Fully onboard with state transitions as event drivers. I understand correctly that this implies that these "predicates" will no longer be published and the DSE has full responsibility to track those internally right?
• Yes for states in conditions and sequences
• The only suggestion I have here is to think about what the cleanest syntax would be. Is it what you proposed (events and predicates or could it be something slightly more verbose

  my_component:
  events:
    states:
      on_load: ...
    predicates:
      is_in_range: ...

  which would be more consistent with the sequence/component syntax, where you can wait for either a state or a predicate event. In a sense, state events and predicate events would be equal since both are just events, and not an event is a state event and a predicate is already an event, if you see what I mean.

Automatic behaviors

• Also a good idea, but I'm wondering again if maybe auto_configure should be under events or top level. Probably rather top level as you suggested.
• I think on_start is still quite important, especially to someone who comes from traditional robotics. We remove that, this person is even more lost :smile:
• Looping of sequences, yes please!

Restrict all names and namespaces to lower_snake_case

Don't see a reason why not.

Positional data in a separate field

• More compact positional data: YES! I just thought about this yesterday when I saw someone using the FE, it was impossible to navigate the YAML file because it was cluttered with this field.
• Positions for conditions and sequences: Probably a good idea, but the syntax should be the same between all blocks so it would have to go into the positions field as well. Also, this syntax can start with components and hardware and then be extended with any other blocks in the FE without being a breaking change.
• Positional data for edges: Probably good for the future :+1:

Display names

I don't mind if we put it into a separate field, it's probably easier to have it as a field of the component/hardware etc. which is fine since we are breaking the syntax anyway.

Application events

Happy that this discussion comes back, it would be great to see this feature finally in the DSE, we had several larger conversations around it already. Certainly the biggest change that the DSE would have to deal with in this epic.

Static frames

This is the part that excited me the least to be frank. Even though I see the need for dealing with frames better, I have my doubts that the suggestions would make it clearer (or maybe I just need to be convinced). It almost feels like this is something that would delay us too much with this epic since I'd rather see a 3D visualization of the robot first. But anyway:

• A dropdown or database of frames would be very nice, but how to create it because most of the frames will come from the URDF.
• You're stating that component and controller description schemas should list frames, but then again, it's the URDF that produces the frames and the frames that they consume depend on the robot that is used so it cannot be statically put into a description.

Metadata

• Yes to syntax, name, description, author
• Package dependencies: How would we do that? Can we look up version of ros packages through ament? The DSE would be responsible for that right?

[eeberhard]

Thank you for the valuable feedback. I agree with you on most points, particularly grouping states, predicates and potentially even auto-events under the events: field.

Display names

I don't mind if we put it into a separate field, it's probably easier to have it as a field of the component/hardware etc. which is fine since we are breaking the syntax anyway.

Can you clarify again which way you prefer or find easier? Is it [A] grouping display names with the actual declaration of components and hardware (what we have now) or [B] moving all display names to a separate field equivalently to how it was proposed for positions?

Static frames

This is the part that excited me the least to be frank

This part is mostly an enhancement, but because it's a large conceptual addition I considered it part of the 2.0 draft. It also the kind of schema addition that can be made before implementing any features to actually use it. If we agree on a conceptual feature and syntax now, we can already add them to the schema even if no components or applications implement or use those features yet.

I'd rather see a 3D visualization of the robot first

I think having a 3D view and having a way of defining and editing static frames in the application file directly is very strongly associated. By adding this concept to the application syntax, we can really inform the design choices for the 3D viewer too. For example, rendering the static frames in 3D next to the robot model, updating their position and orientation interactively, and adding / deleting new frames all in the 3D view. The schema to store these frames should be in place before we implement these interactive features, and we can also have a 3D robot view before this schema is defined, but the two are related and should be designed with respect to each other.

A dropdown or database of frames would be very nice, but how to create it because most of the frames will come from the URDF

I think the frontend can eventually lookup available frames from both the static frames (defined in the application) and any hardware frames (defined in the URDF of the selected hardware), but I should clarify this is just a potential UI feature to the syntax, not a requirement to be ready for the 2.0 draft.

You're stating that component and controller description schemas should list frames, but then again, it's the URDF that produces the frames and the frames that they consume depend on the robot that is used so it cannot be statically put into a description.

What I propose is not that the component description should list the actual frame names they expect. Rather, if a component expects to lookup or broadcast a frame, then the name of that frame should be configurable dynamically. It's exactly the same as signals; if a component looks up a frame that is used as the origin for some trajectory generation, then we currently have some parameter origin_frame: foo, just as we might have had origin_topic: foo for subscription topics in the past. But now we put input signals under inputs: { origin: foo }, so we could similarly treat frames: { origin: foo } as an abstraction for what is essentially just an origin_frame parameter under the hood.

So to summarize, I want component and controller descriptions to declare any dynamic frames that they consume or produce, so that it is easier for us to associate actual frames (and reference frames) with those components when authoring applications.

I think the idea of static frames in the application schema already has a pretty straightforward structure, so (unless there are strong opinions against that idea) the main thing to debate is if / how dynamic frames should be declared in the component and controller descriptions and how they can be associated with real frames and reference frames in the application syntax.

---

I would gladly invite any other thoughts or feedback (@alexialechot, @jjenscodee, @LouisBrunner, @yrh012, @bpapaspyros, @buschbapti).

In the meantime, to continue actioning this epic, I will solidify parts of this proposal in the actual JSON schema for the application under a 2.0 feature branch. As the design proposals become more concrete, I will also publish the bundled schemas under a draft/2-0-0 version so that we can test out how to incorporate it into various parts of backend and frontend and prepare to handle any breaking changes. This way, we can make revisions to the draft before going officially live with 2-0-0.

[domire8]

Can you clarify again which way you prefer or find easier? Is it [A] grouping display names with the actual declaration of components and hardware (what we have now) or [B] moving all display names to a separate field equivalently to how it was proposed for positions?

Sorry yeah not clear from my side. I don't have a strong opinion. What I meant to say is that it wouldn't be so bad to choose the breaking option (for sequences) since we are already breaking the syntax.

because the display name may remain portable with the component even if it is taken out of context of the application.

Because of this, I think it could stay with the "parent", option A

[LouisBrunner]

🤔 CONCEPTUAL CHANGE 🤔

Some changes are conceptual changes even if not technically breaking (old applications may still validate against the new schema, but may not behave as expected).

If your application behaves differently, I reckon it qualifies as a breaking change even though the syntax is the same.

Positional data in a separate field Display names

I am not convinced those should be separated from their "nodes". It doesn't matter too much for automatic production/consumption of the YAML but for human edition it is a pain to have to refer to multiple parts of a file to get the full picture. This pain is also correlated to the size of the file, with massive files being a bit of nightmare.

I think separating "visual" properties into their own sub-object for reading clarity is nice but I am not convinced by having a separate list with matching properties name. I think that makes sense for frames as they could be shared and thus aren't necessarily used in a single place but not necessarily for components, etc.

For display names specifically, maybe that's something we might want to use in logging ourselves so it might also be convenient to have it accessible without too many lookups.

(so I also choose (A))

Package dependencies

Makes sense to me, more checking is always better. Syntax-wise, I think an actual object would be better and simplified your example by avoiding string parsing and inline objects.

Rest

LGTM!

[eeberhard]

Conceptual changes

If your application behaves differently, I reckon it qualifies as a breaking change even though the syntax is the same.

I agree, which is mainly why I want to bring these concepts in to v2.0 while we are already making other breaking changes. I mainly gave them separate labels to provide a bit more structural clarity to the discussion.

Positional data in a separate field

So we have two competing opinions here:

More compact positional data: YES! (@domire8)

I am not convinced those should be separated from their "nodes" (@LouisBrunner)

My main argument for splitting position data into a separate field is making the node declaration itself more compact, portable and relevant to actual runtime behavior.

My main argument against splitting position data into a separate field is the size bloat, usability and maintainability problem that comes from having a "separate list with matching properties name". If you have to write every node name twice, once in the declaration of the node and once in the definition of its position, it adds extra lines and needs to be synchronized.

I would be glad to hear other opinions on this as it's one of the major decisions points in this refactor proposal.

Display names

I also prefer option [A] as described in the previous comments, which is to keep display names with the node declaration.

[jjenscodee]

I think separating position data is fine because I rarely edit it in the YAML part. The position data primarily used for ReactFlow to reload and generate the graph according to our preferences. Typically, we adjust the position data directly from the graph interface.