Affordance as a seed concept for the Interaction vocabulary ?

[maximelefrancois86]

I am strongly in favor of introducing :Affordance in the core vocabulary.

As a rationale the word affordance is already used in the definitions of :Artifact, :Signifier, and :exposesSignifier. It seems like an important concept that would deserve a definition.

:Signifier is Structured data describing (one or multiple) affordances exposed by an artifact to agents.. This suggests that the local range of :describes for :Signifier could be :Affordance, and there could be two more properties:

• :exposedBy that links an :Affordance to the artifact that expose this affordance
• :exposedTo that links an :Affordance to the agents to which it is exposed

[FabienGandon]

Quick question: I thought the idea was not to have hmas:Affordance because affordances were rules/dynamic/relative to an agent and could not be represented objectively and that it was the reason why we focused on Profiles and Signifiers. Did I miss something?

[maximelefrancois86]

If it so happens that I am the one missing something, then it seems I keep forgetting about the rationale for not modeling :Affordances.

It could help me (and potential future users of the ontology) to have this rationale written somewhere, maybe directly in the definition of :Signifier?

[DrLeturc]

I was asking the same question the first time I read the definition of :Signifier. Furthermore, we use the term "affordances" in different classes as e.g.:Artifact : "A resource (or tool) that exposes affordances to agents. Agents use artifacts to achieve their design objectives.".

How do we instantiate (and represent) an :Affordance since there is no class that is explicitly defined in the core ontology.

Is your property :exposedTo means that the agent is currently using an affordance exposed by an artifact ?

I'm also in favor of considering :Affordance as an explicit term in the core vocab.

[DrLeturc]

Initially there is this first discussion about integrating affordance in the core vocab here: "AFFORDANCE [AMBIGUITY ALERT - FROZEN FOR NOW]

• alternative terms: affordance profile, formalized affordance
• definition: An Interaction Affordance is exhibited by an artifact in the environment to inform people and/or autonomous agents (affordances are either observable properties and events, and operations).
• motivating scenario: discover all the capabilities offered by an artefact
• competency question: what are the affordances exposed by an artifact?
• source: /
• note: observable properties and events as well as operations should be defined in the interaction ontology extension

(...)

Questions

• an Affordance is objective and it exists independently of any agent`s perception ?

• an Interaction can be enacted (i.e. an Affordance can be exploited) only if the preconditions that make the Interaction enactable (i.e. the Affordance exploitable) hold. The preconditions hold only if the properties the Artifact and an Agent are appropriate and compatible?

  Possible preconditions:

• Agents: biomechanical properties, protocol binding, content-type, having mental states, current beliefs, etc.

• Artifacts: Structural properties, maximum payload (of a robotic arm), empty gripper (of a robotic arm), forms, links, etc.     ?

• An Affordance does not exist (it is a false Affordance) if the preconditions for exploiting the Affordance can never hold (what about post-conditions/effects?).

• An Affordance may exist but be hidden if the preconditions can hold currently or in the future but an Agent has not discover it?

• Perception of an Affordance: If an Agent (drawn or not by its goals) searches for an affordance that exists and is not hidden, then it can immediately perceive it?

• Enactement of an Interaction: If the Affordance exists, the Agent perceives it and the preconditions hold, the agent can immediately exploit the Affordance (enact the Interaction)?

"

However, in more recent meeting, it seems that affordance should not be in the core vocabulary since it belongs to the perception "inner the agent" and is the way an agent uses a signifier. There is written here :

" On affordances

• affordances should depend on the state of the object and the subjective interpretation of an agent
• a capability of an artifact becomes an affordance as soon as the agent perceive it and if the agent can use it to achieve some goal
• what does it mean to perceive an affordance?
• the affodance changes depending on in what way the agent can use it
• an affordance is something that an agent can use and intends to use
• the affordance is only an affordance in the agent mind
• a way to anchor the interaction / the agent to the object
• by seeing the other agent using the door I understand the affordance; I can't give you the affordance, I can tell you about how I use the affordance

=> affordance might be out of the scope of the interaction ontology "

Furthermore in another doc, here:

"Affordance was introduced by psychologist J. J. Gibson in 1966 to describe the action possibilities offered by the environment to the animal in an attempt to refer simultaneously to “both the environment and the animal in a way that no existing term does” (J. J. Gibson, 1979):

If a gap in a wall has a certain size relative to the size of a person, the gap affords passage (E. J. Gibson, Carroll, & Ferwerda, as cited in E. J. Gibson, 1991; Warren & Whang, 1987).If a stair is a certain proportion of a person’s leg length, it affords climbing (Warren, 1984).

Although there is lack of consensus on a single, uniform, definition of the term, major studies in Affordance Theory view affordances as relations between an animal and its environment that have consequences for behavior (Chemero, 2003; Stoffregen, 2000a; Warren, 1984; cf. Sanders, 1997). This definition aims to capture the nature of affordance, which cannot be simply defined as a property of the environment:

“[...] an affordance is neither an objective property nor a subjective property; or both if you like. An affordance cuts across the dichotomy of subjective-objective and helps us to understand its inadequacy. It is equally a fact of the environment and a fact of behavior. It is both physical and psychical, yet neither. An affordance points both ways, to the environment and to the observer” (J. J. Gibson, 1979).

As summarized by McGrenere and Ho (2000), Gibson specifies three fundamental properties of an affordance (Horton, Chakraborty & Amant, 2012):

• an affordance exists relative to the capabilities of a particular actor;
• the existence of an affordance is independent of the actor’s ability to perceive it;
• an affordance does not change as the needs and goals of the actor change.

Since the 1960s, Affordance Theory has influenced many applied fields such as autonomous robotics (e.g. Şahin et al., 2007), computer vision (e.g. Hassanin, Khan & Tahtali, 2018) and human-computer interaction (e.g. Norman, 2013). Norman, in particular, was interested in perceived affordances in his attempt to design “everyday things” for which the user can easily infer what is afforded to him/her. To this end, he took on the term “signifier” to denote any perceivable cue (deliberate or accidental) that can be interpreted meaningfully to reveal information about affordances. A signifier can majorly enhance the discoverability and interpretability of an affordance - two of the most important characteristics in designing products that can be exploited easily and intuitively (Norman, 2013). In his latest work, Norman shifts completely the designers` attention from affordances to signifiers since the major design goal is to provide ways of “understanding the product or service, some sign of what it is for, what is happening, and what the alternative actions are".

Examples

Typically, a signifier related to an affordance conveys the following information: The type of the affordance, e.g.:

• the blue color of a hyperlink (clickable)
• a “SUBMIT” label on a button (submittable)
• red color on the handle of a train emergency break (in-emergencies-usable)

The means/additional instructions for exploiting the affordance, e.g.

• a “PUSH” sign on a door that affords opening (by-pushing-openable)
• a thumb-shaped notch for opening a lid (with-thumb-openable)

The current status of the object that offers the affordance, e.g.

• a light indicator that a door is locked (not openable) (link to security and regimentation)
• purple color of a hyperlink that has been visited (clickable, already exploited)
• the time until the completion of an action (readable)

The expected status of the object that offers an action affordance that changes the status accordingly, e.g.:

• drawings of a bright lightbulb and a dark lightbulb next to the controls of a lamp that affords switching on and off (switchable)
  • ordered numbers on the rotary switch of a thermostat (value-settable)
• a sign with the range of room numbers that you will encounter if you turn left in a hotel corridor (value-gettable)

Prohibitions and Obligations (link to regulative norms), e.g.:

• street signs that impose prohibition of interaction: the markings on the road convey the affordance of not parking, but a street * sign signifies that parking is prohibited and for how long (not parkable)
  • street signs that impose obligation of interaction: the turn on the road affords turning but a street sign signifies the obligation to turn (must-turnable)

Other:

• a “Kitchen” indicator next to a lamp switch that controls the kitchen lightning (signifies who/what is directly affected)

  • a sign about the location where a door leads to. This signifier implies the effect on the agent, but also possibly ownership or intended usage (link to constitutive norms). Usually, affordance effects are limited to single-step navigation
• expiration date on a food package (edible, affordance life-time)

Quotes

Some of Don Norman`s quotes summarize how the notions of affordance and signifier have been adopted to play a central role in interaction design and human-computer interaction:

Affordances are the possible interactions between people and the environment. Some affordances are perceivable, others are not. (...) Note that some perceived affordances may not be real: they may look like doors or places to push, or an impediment to entry, when in fact they are not.

An affordance is a relationship between the properties of an object and the capabilities of the agent that determine just how the object could possibly be used.

The presence of an affordance is jointly determined by the qualities of the object and the abilities of the agent that is interacting.

This relational definition of affordance gives considerable difficulty to many people. We are used to thinking that properties are associated with objects.

But affordance is not a property. An affordance is a relationship. Whether an affordance exists depends upon the properties of both the object and the agent.

On anti-affordances

Glass affords transparency. At the same time, its physical structure blocks the passage of most physical objects. The blockage of passage can be considered an anti-affordance—the prevention of interaction. To be effective, affordances and anti-affordances have to be discoverable—perceivable.

Two of the most important characteristics of good design are discoverability and understanding (NB: for us it seems interpretability is more intuitive):

Discoverability: Is it possible to even figure out what actions are possible and where and how to perform them?

Understanding: What does it all mean? How is the product supposed to be used? What do all the different controls and settings mean?

Quotes on signifiers:

The term signifier refers to any mark or sound, any perceivable indicator that communicates appropriate behaviour to a person.

Signifiers can be deliberate and intentional, such as the sign PUSH on a door, but they may also be accidental and unintentional, such as our use of the visible trail made by previous people walking through a field or over a snow- covered terrain to determine the best path. Or how we might use the presence or absence of people waiting at a train station to determine whether we have missed the train.

Whatever their nature, planned or accidental, signifiers provide valuable clues as to the nature of the world and of social activities. Perceived affordances often act as signifiers, but they can be ambiguous. Signifiers signal (NB: we prefer signify) things, in particular what actions are possible and how they should be done. Signifiers must be perceivable, else they fail to function. Some signifiers are simply the perceived affordances, such as the handle of a door or the physical structure of a switch. Note that some perceived affordances may not be real: they may look like doors or places to push, or an impediment to entry, when in fact they are not.

Affordances on the Web

"Hypertext (or hypermedia) means the simultaneous presentation of information and controls such that the information becomes the affordance through which the user (or automaton) obtains choices and selects actions. Machines can follow links when they understand the data format and relationship types." Fielding, R. (2008) https://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven.

"(Fielding’s) affordances are, essentially […] the environmental elements of Gibson’s world. Fielding's description also tracks very closely to that of Norman. [...] Applications that rely not just on rules and operations within the code (“in the head” as Norman would say) are capable of recognizing and reacting to affordances in the message itself (similar to Norman’s “in the world”)”Amundsen, M. (2012). From APIs to affordances: a new paradigm for web services. In Proceedings of the Third International Workshop on RESTful Design (pp. 53-60).

“Interaction Affordance: Metadata of a Thing that shows and describes the possible choices to Consumers, thereby suggesting how Consumers may interact with the Thing. There are many types of potential affordances, but W3C WoT defines three types of Interaction Affordances: Properties, Actions, and Events.” W3C Web of Things Thing Description (2020) https://www.w3.org/TR/wot-thing-description/.

Affordances and Signifiers in Hypermedia MAS

In a Hypermedia MAS, autonomous agents interact with artifacts in their environment in a uniform manner through hypermedia. Although hypermedia reduces coupling in interactions between agents and artifacts, addressing the discoverability and interaction guidance of such heterogeneous and dynamic components within a Web-scale environment remains a challenge. We hypothesize that affordances and signifiers will give environment designers and artifact designers the tools to design environments and artifacts that present cues about which agent abilities and artifact capabilities a) are compatible with each other, and b) are relevant to enacting an interaction. For example, an agent would perceive only those signifiers that are relevant to it and environment and artifact designers could choose how salient the signifiers should be for different types of agents.

Affordances can provide a means to cope with the heterogeneity and evolution of agentsabilities and artifacts capabilities. Representing the complementarity between agents and artifacts through affordances can minimize the load while searching for action possibilities by limiting the set of available signifiers to one that maps to the affordances that can actually be exploited by an agent. Representations of affordances could additionally guide the evolution and behavior of agents, by enabling them to reason and act towards acquiring new abilities or delegating their objectives to an agent that already has the required abilities to exploit an affordance. Finally, defining affordances in terms of abilities and capabilities, instead of specific agent and artifact instances, may decouple further agents from their environment.

Signifiers can support agents to achieve more flexible behaviour at run-time or the filtering of affordances based on selected criteria. The semantics of signifiers can enhance the interpretability of affordances, thus helping agents to reason on cues about affordances and engage in purposeful interactions towards their objectives. Apart from purely informative semantics, signifiers may carry normative meanings that render the regulatory nature of interactions. Signifiers with social semantics can additionally enable the coordination of agents through stigmergy. Finally, the salience of a signifier may change dynamically based on objective values (e.g. based on the capabilities or the internal state of an artifact that is used in an organization) or subjective values (e.g. based on the abilities and the objectives of an agent). By differentiating interaction cues among parallel receivers, interactions can be guided in a way that is specific to an agent and its context. Since agents are autonomous, a signifier can be interpreted subjectively by an agent: the semantics and salience of a signifier may or may not have consequences for the agent's behaviour, or even have consequences that are unobserved or unintended from the standpoint of artifact and environment designers. For example, agents of one organization may interpret differently an affordance than the agents of another organization. At the same time, an agent may interpret differently an affordance than any other agent does.

Positioning towards direct perception

Based on Affordance Theory, affordance perception is direct: if an agent (drawn or not by its objectives) looks for an affordance that exists and is not hidden, then it can immediately perceive it. Agents might be required to interpret signifiers in order to infer information about affordances (e.g., by finding a formal alignment to a known vocabulary); but they do not need to reason across affordances to perceive and understand one.

Signifiers versus affordances

Signifiers are inherently present in the environment without the need of an agent to be present. In contrast, the value of an affordance depends on the agent (e.g. on the agent`s abilities and internal state) that is present in the environment. Signifiers have objective meanings and convey information about an affordance. Specifically, signifiers refer to any perceivable indicator that conveys information about proper behavior to agents. This objective is very much aligned with the process of semantic annotation which aims in increasing content discoverability, interpretability and reusability.

Signifiers as Web Annotations

An annotation is a Web Resource that is considered to be a set of connected resources, typically including a body and target, and conveys that the body is related to the target.

Figure 1: Selected concepts of W3C Annotation Vocabulary (Sanderson et al., 2017)

Advantages of using the W3C Web Annotation Model

Annotations convey meaning about a resource without modifying the resource itself. Annotations enable the decentralized augmentation of Web resources -> enables the agent-driven content enrichment of Web resources that adds a social layer on top existing resources and allows for opinion-sharing and knowledge sharing among agents.

Annotations can be of different types or motivated by different contexts, thus they may affect agents’ behaviours in different ways, e.g.:

• to inform (e.g. commenting)
• to draw attention (e.g. highlighting)
• to regulate (e.g. moderating)
• to promote social interactions (e.g. questioning, replying)

The model provides mechanisms for describing the provenance and motivation of publishing/updating an Annotation -> improves transparency. The model is a W3C Recommendation that allows for a shared format of Annotations -> already enables the implementation of mechanisms on top of uniform representations of Annotation (e.g. for their aggregation, evaluation, agent-artifact matching processes etc.).

"

[DrLeturc]

@danaivach : Can you give us your point of view (because this is your work I quoted) ? : )

[vcharpenay]

@DrLeturc, you almost quoted the report on affordances in extenso; could you highlight the relevant parts that motivate the choice of not giving a definition of Affordance?

The argument seems to be the following (quote by you, I think):

affordance should not be in the core vocabulary since it belongs to the perception "inner the agent" and is the way an agent uses a signifier

but I see a certain contradiction with parts of the report saying e.g. that

the existence of an affordance is independent of the actor’s ability to perceive it (p. 2)

or that

Some affordances are perceivable, others are not. (p. 4)

If affordance are (sometimes) perceivable, if they have an existence of their own, they don't belong to the inner of an agent.

[vcharpenay]

Moreover, the report gives scenarios involving a search among available affordances (p. 8-9) and an open question:

Q3: How can we represent affordances in Hypermedia MAS?

An initial answer is given in the report. From the 2003 essential publication "How Shall Affordances be Refined? Four Perspectives", a similar* suggestion is made by Chemero (an affordance is a statement of the form afford-<behavior>(<ability of an agent>, <feature of the environment>)).

So, it seems to me that affordances would have, at some point, a(n RDF) representation. What are your opinion on this (INRIA, St. Gallen)?

(*) in Chemero's proposal, however, an affordance doesn't depend on the internal state of the agent (nor on the internal state of the artifact, it seems to me).

[FabienGandon]

I believe it is really important that @andreiciortea , @danaivach and @smnmyr check this point and what I am going to say here.

If I remember well we stopped using the term affordance precisely because (1) it was ambiguous and it was giving the wrong impression of talking about the same thing in very different contexts and (2) the definition we were converging on was indeed dependent on "the agent's abilities and internal state" and therefore wasn't part of the externalized knowledge exchanged and targetted by the ontology.

The introduction of the term Signifier was specifically for capturing the shared "perceivable aspects" without using the term Affordance that has been found to ambiguous and overloaded.

my 2 cents,

[andreiciortea]

@FabienGandon agreed on my side, great concise summary

We might need to reconsider an appropriation of the term "affordance" for the interaction vocabulary if we have use cases that require the term to represent signifiers (e.g., what is a given signifier for), but I think we can and should go without given the ambiguity of the term "affordance" and our current understanding of affordance as a relational notion (e.g., we could focus on goals if we need to model signifiers rather than affordances). I'll let @danaivach and @smnmyr jump in for more details.

[smnmyr]

I also agree with @FabienGandon .

@danaivach - you are at the core of this definition, also with the current paper. Please weigh in here.

Based on @danaivach 's input, I propose that we revisit all occurrences of "affordance" in the current definitions and replace them by signifiers where applicable.

[danaivach]

I believed this was resolved. I'm sorry.. On our side, we suggest:

• to use Signifier as a seed concept for the Interaction vocabulary and
• to do not use Affordance as a term in the ontology.

By using Signifier , we can align with the HCI approach on design of interaction cues. This implies that we enable taking an agent-centered approach in designing signifiers, i.e. considering the agent-artifact complementarity when describing signifiers. The latter shows that we are inspired by affordance theory, and we can use the concept of "affordance" when discussing or encouraging the use of signifiers to describe part of the relational aspects of affordances. It does not impose however that signifiers necessarily describe all relational aspects of affordances. As a result, signifiers could be used to describe interaction metadata that come a bit closer/are inspired by an affordance definition (e.g. incl. abilities of agents based on Chemero's definition which Victor mentioned), as well as metadata provided e.g. in td:InteractionAffordance.

By not using Affordance as a term in the ontology (at least for now), we avoid overloading a somewhat elusive and already overloaded term. Representing affordances in the internals of agents has been commonly criticized as going against affordance theory (see Gibsonian affordances for roboticists). Additionally, more generally representing affordances based on Chemero's definition would impose representing affordances in a circular way, which I do not know how useful can be for our systems: E.g. describing an affordance in relation to an agent's ability and the environment situation <affordance1> = <ability1, situation1> would impose describing the agent's ability in relation to the affordance <ability1> = <affordance1, behavior1> (see Gibsonian affordances for roboticists and Complexity, Hypersets, and the Ecological Perspective on Perception-Action) (the same applies for Turvey's affordance definition discussed in the same papers).

[maximelefrancois86]

MINES Saint-Etienne:

If the term 'affordance' is disputed and needs to be avoided in the core ontology, we would request at least a different definition for :Signifier that does not use this word.

We would like to request a scenarios + examples that illustrate how you wish to use the concept Signifier.
Maybe with web annotations as in @danaivach 's report ?

[vcharpenay]

The general discussion on affordances shifted to a more focused one on signifiers. This discussion on signifiers goes to #82.