How will the user communicate with our system? How should the system respond?

[maddalihanumateja]

6 got me thinking about how a language for communication via objects or pictures would look like in our project. How have past experiments approached this / what did the participants (maybe participants with dementia) do when provided with a system similar to what I'm building? How does the system provide intelligent assistance (assuming the absence of a human guide)?

• Question 1: Will the users need an unstructured way of interacting with the system? One at a time/ multiple at a time. What is our system to do if the subject of the instruction is presented first? What it is an object / action / adjective first? We need ai to understand and plan best course of action in this case.

  • Case 1: A user present his brother's photo/associated object as the first input. The system determines what information is missing in a "person first" template and says "This is your brother right? Would you like to do something that involves him? (call him, or send him something?)"
  • Case 2: A user presents an action photo/associated object as the first input. The system determines what information is missing in an "action first" template and says "Do you want to send something"
  • Case 3: A user presents a freeform input photo/associated object as the first input. The system determines what information is missing in a "freeform input first" template and says "This is a nice something (painting). Would you like to share this with someone?"
  • Case 4: A user presents a freeform input photo/associated object and brother's photo/associated object as the inputs. The system determines what information is missing in a "multiple inputs" template and says "This is a nice something (painting). Would you like to share this with you brother?"

• Question 2: Can we force our system to only allow images representing a subject first, then verb, and then object? Would this be ok with the user? For example: The system only searches for photos/associated objects for people and then asks the user what he wants to involve the detected person in.

We can also think of what form these templates take (maybe we can look at what linguistics and AI-assistants or chatbots do and adapt it to this setting).

[maddalihanumateja]

I've encountered templates as a way to program smart cards in our assisted living reference paper. This is different from the way that I look at these sharing images or objects. I want an assistive AI to be able to figure out what information could be missing/additional/optional and suggest/prompt the user to present this missing/ additional/ optional information.

This is the related image from the paper:

A similar message (SMS) composition grid was found in this reference paper. In this paper the researchers have built a TUI (the example use case they've mentioned is older adults with some special needs). There is a message composition process where "in the first step user selects message core (e.g. “Please bring me”), then time (e.g. “Today”), after that place is selected (e.g. “Pharmacy”) and object at the end (e.g. “Medicines”). Result of selections is a string with value “Please bring me medicines from the pharmacy today.” which is semantically organized by the application inside the NFC probe device."

[maddalihanumateja]

What I'm hoping to implement however is an assistive process which doesn't force the user in a specific structured form of communication using the TUI objects. I want the "assistant" to adapt to the user's input (subject first, object first, ...) just as in a normal conversation with an assistant. I'm also assuming that this AI assistant can allow a single user to interact with the interface independently. The references I cited previously are old papers. I need to search for newer ones that mighthave already implemented something similar to what I'm thinking.

[maddalihanumateja]

Another reference for this (that I remember from a comp. cognitive science course) is the chapter on concepts in Thagard's "Mind" textbook. Where we can think of our basic SharingImage types as concepts that can be described by certain slots and rules for how to fill in those slots. This process may involve multiple people in our case so we could figure out where an AI/human helper can fit into this.

[maddalihanumateja]

While trying to resolve #22 I started drawing graphs showing relations between SharingImage objects which looked similar to what you would get from <subject, predicate, object> triplets. Also remembered RDF triples from the data modeling lit survey I did a couple of years back. Tried searching for RDF databases that already had some concepts defined for our general set of actions. I think it'll be easier if I just implement something similar on my own (for a small set of actions).