"interrupt the current system action"

[kuaduer]

Hi there. The FLoReS dialogue management engine looks very interesting. I have a question about its interruption-handling. In the online document, it says we can define a system edge/act as "an interruptible speech act", which can be interrupted if the user says something that triggers an "interrupt request". Also, it says we can define the effect of a node as "a request to interrupt the current system action". It seems to me this is not demonstrated in the given examples. I tried to modify the CakeVendor example by defining some interruptible system actions using the "isystem" keyword but still don't know how to make it work. Could you please give me a hint or a simple example to show how the interruption-handling works? Thanks a lot! -Changsong

[fmorbini]

interruption work when the system drives a virtual human (or at least a text to speech system). The tts must provide information to the dialog manager when a certain utterance ends playing. Check out the uses of the DMInterruptionRequest, for example when issued because of the DMConfig.getUserAlwaysInterrupts and DialogueOperatorEffect.isInterrupt.

what functionality are you looking to implement?

[fmorbini]

the #action: interrupt should be used in a daemon like network that is triggered for example by a system initiative condition on the length of time the user has been speaking. Tracking how long the user has been speaking requires having your animation system communicating start and stop events (for example ict's vr system of events for virtual humans).

[kuaduer]

Thanks for your quick reply. What I am trying to implement is this: suppose the system is executing an action that has a long duration, for example, reading tomorrow's weather forecast to the user. Then in the middle, the system detects that the user just had a load sneeze. We want the system to suspend what it is saying (i.e., the weather forecast) and respond to the sneeze right away ("Bless you! Are you OK?"). So the current weather-forecast subdialogue should be interrupted by the sneeze event and a sneeze-bless subdialogue should be activated. After the sneeze-bless subdialogue is done, the system should be able to resume the weather-forecast subdialogue from where it was interrupted. We tried the "UserAlwaysInterrupts" setting, and it seems to let user's input always interrupt the system's action, no matter what the user input and the system action are. What if I only want some system actions to be interruptible but not all, and only some specific user inputs/events can interrupt the system but not every input always interrupt the system? Thanks.

[fmorbini]

yes, we had exactly that (well not for the sneeze but for speech) in http://ict.usc.edu/prototypes/simsensei/ i'll post a piece of the policy that did that tonight. I'm not working anymore at ICT, you can reach out to the NL group there to get more info if you like.

[kuaduer]

Great! Looking forward to the example. Thanks a lot for your help.

[fmorbini]

these where the 2 networks in charge of the interruption policy. GIVEUPFLOOR is a macro with some system lines to say (like "i'm sorry, go ahead", things like that). userSpokeForSeconds, ssytemNowSpeaking, nowSpeaking are all special variable set automatically by the dm depending on the info coming from the animation and asr systems.

daemon network interruption {

var: interruptedSystemEvent=null

#var: interruptionTime=timeSinceStart
#entrance condition: system initiative
#condition: and(userSpokeForSeconds>1,systemNowSpeaking,isInterruptible(),nowSpeaking,forcedstate!='start')

DO
#action: interruptedSystemEvent=systemEvent
#action: interruptInProgress=true
#action: ++(interruptionCount)
#action: interrupt
#send: interrupt
#action: swapout

if (false)
{
    DO
    #reentrance option:
    #condition: !interruptInProgress
}

} network dointerrupt {

topic: interrupt

#entrance condition: current NLU speech act = interrupt
system: GIVEUPFLOOR
#goal: default
#action: interruptInProgress=false

}

[kuaduer]

Hi Fabrizio, thanks a lot for the example. I made my own one based on your example and it worked as expected. This example actually reveals some more advanced features of FLoReS that weren't described in the online documentation. Thanks for your great help. I have some more questions about your example, could you please take a look: Q1: if I define the interruption network exactly as yours, it will only work once (the first time). The way that I made it work repetitively is to remove the #action: swapout and the if block below it. Could you please tell me the rationale of using the swapout and reentrance commands here? Why it is not working for me? Q2: I am confused about the if(false) {DO ...} block. Doesn't if(false) mean this block will never be executed? Thanks. -Changsong

[fmorbini]

swapout puts that action in the dormant state. A dormant action can be awoken only by a reentrance condition (or a restart). The swapout was there (if i remember right) to avoid having the same network fire over and over. An action when it's in dormant state it remembers its own state.

the if (false) is a trick to skip that code from any other entry point. Networks are fsm that can have multiple entry points. For example, if you have 2 entry points A and B and A is before B, if you enter through A you'll pass through B. if you want to do specific updates only when entering through B then you can enclose the entry point B in a if (false) together with the updates specific to B. In this specific case the false doesn't do anything specific and can be removed.

if you put the RewardPolicy logger in debug mode, the code will generate gdl files of each network that you can open using the aisee software. Unfortunately it's not available anymore as a standalone product. But you can still get it from the web archive: http://web.archive.org/web/20140628235908/https://www.absint.com/aisee/download/

if you also run the RewardDM with the logger in debug mode it will generate gdl graph of the search with info in its nodes. There are a few bugs in the search with local vars that have not been ported to the jmnl version unfortunately.

[kuaduer]

Thanks! This was a very helpful discussion. -Changsong

[kuaduer]

Hi, Fabrizio, I tried to put the RewardPolicy logger in debug mode, i.e., setting src/log4j.properties to log4j.logger.edu.usc.ict.nl.dm.reward.model.RewardPolicy=debug. However, got the error when running the chatInterface: INFO 11:19:56.876 [Thread-4 ] [NLBusBase ] DONE starting new NLG for character: CakeVendor INFO 11:19:56.876 [WT-EventQueue-0] [NLBusBase ] edu.usc.ict.nl.bus.NLBus Ready. java.lang.NullPointerException at edu.usc.ict.nl.ui.chat.ChatInterface.startDefaultCharacter(ChatInterface.java:593) at edu.usc.ict.nl.ui.chat.ChatInterface$21.run(ChatInterface.java:1259) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80) at java.awt.EventQueue.dispatchEvent(EventQueue.java:726) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Any hint? Thanks! -Changsong

[fmorbini]

line 593 doesn't seem to be a valid line, any other exceptions you see? any local changes?

[kuaduer]

Hi, I was able to generate the .gdl file for a text policy by running RewardPolicy.main(). However, I found the .gdl graph has too many edges and is not very informative. For example, the graph for the CakeVendor policy is shown below. It looks very different from the pictures shown in the online document. Is it because I am using the demo version of the aiSee software, or this is actually what a policy looks like? Thanks.

[fmorbini]

Hi, no that image is not the policy, that is an approximation of the dependencies between operators. The dm can use that info (if a particular config flag is set) to speed up the search (as it becomes an approximated search).

the gdl of the policy is obtained by setting the RewardPolicy to debug in src/log4j.properties config file: log4j.logger.edu.usc.ict.nl.dm.reward.model.RewardPolicy=debug see line 313 and 315 in RewardPolicy.java if (logger.isDebugEnabled()) o.toGDLGraph("policy_"+policyID+"operator"+o.getName()+".gdl"); ...

if the RewardPolicy logger is in debug mode, then it'll generate those graphs.

let me know if you have further questions.

[fmorbini]

oh, just set the debug level and then start the chat interface or whatever other method you use to load the various policies you have. It'll generate two files for each operator. Look at the post-processed one as that is the one that will be used.

[kuaduer]

Thanks. When setting the RewardPolicy to debug in src/log4j.properties, running chat interface gets some errors and couldn't load any character (the same thing happens even for the original github code). I now can load a policy file and generate the gdl for each operator, however, the generated operator graphs seem to always only have a single node and nothing else, e.g., _graph: {display_edgelabels: yes node: { shape: box title: "364" label: "caketype"} } Not sure what is wrong here. Nevertheless, we found that the generated xml format of the policy seem to be correct and is very informative. We should be able to easily convert the xml format to something (e.g., dot) that can be visualized.