Agent jerky head movements in games

@mbehrooz I suspect this is a non-trivial logical change, so please look at other things first. Now that Lazlo has smoothed out the gaze motion, I can see more clearly that the jerkiness in agent is due to the fact that she always first looks straight, before looking left or right.

If you can see an obvious way to optimize this, great. If not, we can live with it. There are far bigger fish to fry ;-)

Sure, I just remembered that I noticed it too and it was not like this before recent 'apparent' agent changes in graphics (about a month ago). So I was assuming it is a bug in graphics being worked on.

I think the gaze command is not set to be straight in the middle in games (say going from thinking to board). I keep sending gaze commands for thinking and then start (keeping) sending gaze commands for board with no gaze related commands in the middle...

I will think about it more if there is anything to make it efficient in the game side :)

I think if you watch it is pretty clear that when she finishes thinking, she first looks straight ahead and the decides she needs to look at the board. Maybe you can check first for whether it is time to look at the board?

I totally see the picture it draws when watching it, but I saw the code and I sent the board gaze command straight from when agent is in thinking gaze and there is no intermediate gaze state/command sent by Java. In schema interval this is what happens:

Send thinking gaze ... Send thinking gaze ... Send thinking gaze ... Send board gaze ... Send board gaze ...

So it looks like it is out of control in Java. The fact that it was not doing this jerky move before agent was updated confirms it too.

I could take a loot VS code if you wanted. Thanks a lot :)

I am not convinced. Why don't you look at the console log trace of the messages to see for sure. I think I saw some gaze(0,0)'s in there! Btw, you need to set TcpConnection.STARTS_WITH to 0 in order to stop it from suppressing multiple gaze commands in a row (unless they are identical).

Ok definitely, I will double check this more deeply from logs and report back (after my class).

Actually, don't right now. It is not top priority.

On Thu, Feb 6, 2014 at 9:41 PM, mbehrooz notifications@github.com wrote:

Ok definitely, I will double check this more deeply from logs and report back.

— Reply to this email directly or view it on GitHubhttps://github.com/always-on/always/issues/269#issuecomment-34368399 .

Dr. Charles Rich, Professor, Computer Science Departmenthttp://www.cs.wpi.edu Interactive Media and Game Development Programhttp://www.wpi.edu/academics/imgd/ Robotics Engineering Program http://www.wpi.edu/academics/robotics/ Worcester Polytechnic Institute, Fuller Laboratories B25b 100 Institute Road, Worcester, MA 01609-2280

Email: rich@wpi.edu Phone: 508-831-5945 Fax: 508-831-5776 Home: http://www.cs.wpi.edu/~rich

Just a quick log look:

As I see below, in the start of the TTT, agent plays and there is a thinking gaze and board gaze and then a user gaze (which is sent only once for the cases when face tracking is not there.)

So between the first and second gaze there is no 0,0.

The "useronce" which sends only one gaze to user can be removed and the functionality would be the same if the facetracking is there.

* Sent {"msg_type":"gaze","msg_body":{"vertical":"0.85","horizontal":"1.00125"}} Sent {"msg_type":"show_menu","msg_body":{"menus":[],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[]"}} * Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.agent_cell","msg_body":{"cellNum":5}} Sent {"msg_type":"ticta-- 1 more Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"speech","msg_body":{"text":""}} Received {"msg_type":"done","msg_body":{"action":"speech","data":""}} Sent {"msg_type":"speec-- 1 more *\ Sent {"msg_type":"gaze","msg_body":{"vertical":"0.0","horizontal":"0.0"}}

Yours, Morteza

First, let's have the default config be to assume there is face tracking.

Aha, so you are saying that it is face tracking that is causing the snap to the middle between thinking and looking at the board? That would make sense if on some call to the schema's run method, it returned NullBehavior, which would free the gaze resource and alllow the face tracking to start.

That is the correct behavior when the agent is waiting for the user to respond to a menu, but in the case of "think and move" (which is really all one action), there should be no moment when the schema returns NullBehavior.

Does this make sense (and correspond to how you see the system working? If not, let's talk on the phone.

Btw, this applies to all games.

Exactly, but there is no schema run that would return null as far as I see. I printed out all the gaze values which correspond one to one to the respective behaviors that are sent. This is TTT and one agent turn. (each are 1 schema run, then no null behavior sent in between any of them).

thinking thinking thinking thinking thinking board board board board board sayandgaze sayandgaze sayandgaze sayandgaze sayandgaze sayandgaze

Btw, since sayandgaze included GazeBehavior.USER, it will prevent face tracking from operating.

The definitive way to trace the behaviors is to add a print in CandidateBehaviorsContainer.add(). That's the only way to be sure that TTTSchema is asking for the gaze resource on every tick between think and board...

Ok sure, well in that case, if I am not mistaken, the frequency of that class is higher than the schema? So between ANY two commands from ANY one plugin there will be null behavior?

This also the log from that method. The simple() is sent always between any two commands.

Simple(GAZE(java.awt.Point[x=-18,y=-900])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=-18,y=-900])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=-18,y=-900])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=-18,y=-900])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=-18,y=-900])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=338,y=1320]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}}Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=338,y=1320])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=338,y=1320])) Simple() Simple() Simple() Simple(GAZE(java.awt.Point[x=338,y=1320])) Simple() Simple(FaceTrack)

You should also print out which schema is making each behavior request, and it will be very clear what is or is not going on.

The priority of face tracking should be lower than the priority of any activity like the game, but that is irrelevant here, since if the game is not using the GAZE resource, then face tracking is free to use it.

Ok, here it is, the part that it transits from thinking to board gaze. I am not really sure what to make of this though.

edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=338,y=1320]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}}

edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple() edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@2675da69 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@78e1b737 >> Simple(FaceTrack) edu.wpi.always.ttt.TTTSchema@654dbbf6 >> Simple()

I am responding to the email here, Thanks for looking at it. That gaze number must be one of standard Gaze Behavior direction as I am not using any other, translated into numbers, I am guessing board (Plugin).

Here is the full log:

Agent type = Unity User ontology file: ......\user\User.owl Loading user model from: ......\user\User.owl User name: Diane Ferguson Using closeness = Companion Starting DiscoRT... Starting Collaboration Manager... Loaded plugin: edu.wpi.always.ttt.TTTPlugin@fffcf51 edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() Always running...

Sent {"msg_type":"speech","msg_body":{"text":" hm <CAMERA ZOOM=\"1.6\"/>"}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"true"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest))

New focus: edu.wpi.always.ttt.TTTSchema@290f0613 Sent {"msg_type":"speech","msg_body":{"text":"Do you want to play the first move or should I?"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest))

Sent {"msg_type":"show_menu","msg_body":{"menus":["Let me play first","You go ahead"],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}} Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest))

Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple()

Received {"msg_type":"menu_selected","msg_body":{"text":"You go ahead"}} Sent {"msgtype":"show-- 1 more Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"false"}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"0.85","horizontal":"1.00125"}} Sent {"msg_type":"show_menu","msg_body":{"menus":[],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[]"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple()

Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}}edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple()

Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.agent_cell","msg_body":{"cellNum":5}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple()

Sent {"msg_type":"ticta-- 1 more Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell "))

Sent {"msg_type":"speech","msg_body":{"text":""}} Received {"msg_type":"done","msg_body":{"action":"speech","data":""}} Sent {"msg_type":"gaze","msg_body":{"vertical":"0.0","horizontal":"0.0"}} Sent {"msg_type":"speech","msg_body":{"text":" I got the golden cell "}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell "))

Received {"msg_type":"done","msg_body":{"action":"speech","data":" I got the golden cell "}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple()

Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"true"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest))

New focus: edu.wpi.always.ttt.TTTSchema@290f0613 Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Yes, that's the best place to start\",\r\n \"Yes, now what are you going to do?\",\r\n \"Yes, but I can still win\"\r\n]"}} Sent {"msg_type":"show_menu","msg_body":{"menus":["Yes, that's the best place to start","Yes, now what are you going to do?","Yes, but I can still win"],"twoColumn":false,"extension":false}}edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.ttt.TTTSchema@290f0613 >> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05 >> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple()

You're right. There is no gaze message being sent to client between thinking and board gaze. I will take this up up w Lazlo re animation.

Tx, Chuck

Sent from my Droid...

Dr. Charles Rich, Professor, Computer Science Department Interactive Media and Game Development Program Robotics Engineering Program Worcester Polytechnic Institute, Fuller Laboratories B25b 100 Institute Road, Worcester, MA 01609-2280

Email: rich@wpi.edu Phone: 508-831-5945 Fax: 508-831-5776 Home: http://www.cs.wpi.edu/~rich On Feb 9, 2014 11:14 PM, "Morteza Behrooz" notifications@github.com wrote:

I am responding to the email here, Thanks for looking at it. That gaze number must be one of standard Gaze Behavior direction as I am not using any other, translated into numbers, I am guessing board (Plugin).

Here is the full log:

Agent type = Unity User ontology file: ......\user\User.owl Loading user model from: ......\user\User.owl User name: Diane Ferguson Using closeness = Companion Starting DiscoRT... Starting Collaboration Manager... Loaded plugin: edu.wpi.always.ttt.TTTPlugin@fffcf51https://github.com/edu.wpi.always.ttt.TTTPlugin/always/commit/fffcf51 edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() Always running...

Sent {"msg_type":"speech","msg_body":{"text":" hm "}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"true"}}edu.wpi.always.ttt.TTTSchema@290f0613>> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest))

New focus: edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613 Sent {"msg_type":"speech","msg_body":{"text":"Do you want to play the first move or should I?"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Constraints(SPEECH("Do you want to play the first move or should I?"), MENU(Let me play first, You go ahead)), Simple(FocusRequest))

Sent {"msg_type":"show_menu","msg_body":{"menus":["Let me play first","You go ahead"],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}} Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest))

Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(Null, MENU(Let me play first, You go ahead)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple()

Received {"msg_type":"menu_selected","msg_body":{"text":"You go ahead"}} Sent {"msgtype":"show-- 1 more Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"false"}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}}edu.wpi.always.ttt.TTTSchema@290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"0.85","horizontal":"1.00125"}} Sent {"msg_type":"show_menu","msg_body":{"menus":[],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[]"}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple()

Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}}edu.wpi.always.ttt.TTTSchema@290f0613 >> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=-18,y=-900])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320]))

Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}}edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple()

Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.agent_cell","msg_body":{"cellNum":5}}edu.wpi.always.ttt.TTTSchema@290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple(GAZE(java.awt.Point[x=338,y=1320])) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple()

Sent {"msg_type":"ticta-- 1 more Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}}edu.wpi.always.ttt.TTTSchema@290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell "))

Sent {"msg_type":"speech","msg_body":{"text":""}} Received {"msg_type":"done","msg_body":{"action":"speech","data":""}} Sent {"msg_type":"gaze","msg_body":{"vertical":"0.0","horizontal":"0.0"}} Sent {"msg_type":"speech","msg_body":{"text":" I got the golden cell "}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce >> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell "))

Received {"msg_type":"done","msg_body":{"action":"speech","data":" I got the golden cell "}}edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Constraints(GAZE(java.awt.Point[x=160,y=120]), SPEECH(""), SPEECH(" I got the golden cell ")) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple()

Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"true"}}edu.wpi.always.ttt.TTTSchema@290f0613>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest))

New focus: edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613 Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Yes, that's the best place to start\",\r\n \"Yes, now what are you going to do?\",\r\n \"Yes, but I can still win\"\r\n]"}} Sent {"msg_type":"show_menu","msg_body":{"menus":["Yes, that's the best place to start","Yes, now what are you going to do?","Yes, but I can still win"],"twoColumn":false,"extension":false}}edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple() edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.ttt.TTTSchema@290f061https://github.com/edu.wpi.always.ttt.TTTSchema/always/commit/290f0613>> Sequence(Simple(MENU(Yes, that's the best place to start, Yes, now what are you going to do?, Yes, but I can still win)), Simple(FocusRequest)) edu.wpi.always.cm.schemas.FaceTrackerSchema@5a41ca0https://github.com/edu.wpi.always.cm.schemas.FaceTrackerSchema/always/commit/5a41ca05>> Simple() edu.wpi.always.cm.schemas.MovementTrackerSchema@1eb23fchttps://github.com/edu.wpi.always.cm.schemas.MovementTrackerSchema/always/commit/1eb23fce>> Simple()

— Reply to this email directly or view it on GitHubhttps://github.com/always-on/always/issues/269#issuecomment-34588923 .

@lring I am transferring this issue to you, since after investigation it appears to be a bug in the agent's head/gaze animation.

(@mshayganfar this might also cause some problems with face tracking---depends on what is actually going on)

The issue is easy to see if you simply run the TTT plugin standalone and let the agent go first. She gazes to her up/left for thinking and then to her bottom/right to look at the board (and make her move). In between, she very clearly snaps to the straight ahead position, on the way between these two positions, instead of smoothly turning.

If you look at the log trace below, you will see there is no gaze command being sent between the thinking and the board gaze.

-CR

Sent {"msg_type":"speech","msg_body":{"text":" hm "}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}} Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"true"}} New focus: edu.wpi.always.ttt.TTTSchema@290f061 Sent {"msg_type":"speech","msg_body":{"text":"Do you want to play the first move or should I?"}} Sent {"msg_type":"show_menu","msg_body":{"menus":["Let me play first","You go ahead"],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}} Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[\r\n \"Let me play first\",\r\n \"You go ahead\"\r\n]"}} Received {"msg_type":"menu_selected","msg_body":{"text":"You go ahead"}} Sent {"msgtype":"show-- 1 more Sent {"msg_type":"tictactoe.playability","msg_body":{"value":"false"}} Sent {"msg_type":"start_plugin","msg_body":{"name":"tictactoe","instance_reuse_mode":"Reuse"}}

HERE IS THINKING GAZE

Sent {"msg_type":"gaze","msg_body":{"vertical":"0.85","horizontal":"1.00125"}}

Sent {"msg_type":"show_menu","msg_body":{"menus":[],"twoColumn":false,"extension":false}} Received {"msg_type":"done","msg_body":{"action":"show_menu","data":"[]"}} Received {"msg_type":"done","msg_body":{"action":"speech","data":"Do you want to play the first move or should I?"}}

HERE IS LOOKING AT BOARD GAZE (NO OTHER GAZE BETWEEN!)

Sent {"msg_type":"gaze","msg_body":{"vertical":"-1.0","horizontal":"-1.00125"}}

Quick update, this does seem to be a bug in the animation system, will see what I can do about it.

@charlesrich @mshayganfar I have been working with our animator on fixing this problem but when we test it in the real system it still seems off. We have fixed the jerkiness of the motion but the mapping is off.

Would it be possible to send us a drawing showing where you expect the user is sitting in relationship to the computer and the values you think you will be outputting at those positions?

That way we can adjust the animation values accordingly.

Hi Lazlo, I am glad you fixed the jerkiness, but I am not sure why you are concerned with where the user is sitting. It is up to Mohammad's face-tracking code to do the mapping from the camera image/position to the gaze angle, based partly on geometry and partly on (informal) experimentation with user perceptions (the goal being perceived engagement). Furthermore, the geometry between the camera and the avatar changes when the agent is in the large versus small configuration (Mohammad's code deals with all of this).

So all we need from the animation is for (0,0) to be straight ahead (in the reference frame of the agent) and the gaze to scale linearly between there and the maximum values (+-5, I think?).

Furthermore, since Mohammad just spent a lot of time re-calibrating the relevant transformations, it would be nice if the animation range stayed the same as before.

Thanks, -Chuck

Hi Chuck,

We are trying to adjust the animations so that Mohammad does not have to re-calibrate the values.

The question we have is about the value range coming from the face capture system since it does not seem to be a simple -1 to 1 range from what we can see.

According to the original specifications we discussed, the gaze range goes between -1 to 1 for both horizontal and vertical, in which the the |1| values represent looking off into the distance (I.E. Thinking). When we test the system however, the range seems to be between -.3 to .6.

If we could just get the exact specification of what the face tracking system is returning we can make sure that any changes we make to the animation system will still work with it.

--Lazlo

Hi Lazlo, I think we are still talking past each other. I can only try to repeat what I said a different way: the animation should not be tailored to the face tracking.

We would like the animation to remain just as it was with (0,0) being straight ahead and +-1 being whatever was previously her maximum head turn in each direction (and linearly scaled in between). All of these are in the agent's frame of reference.

As I said, how much we actually want to turn her head depends on tuning to see what looks best (and also depends on where her eyes are relative to the camera, which changes between the large/small configurations). We may indeed not use the full range of her possible motion.

I am leaving the country tomorrow morning, so I hope this is clear enough.

-Chuck

On Wed, Feb 12, 2014 at 8:29 PM, lring notifications@github.com wrote:

Hi Chuck,

We are trying to adjust the animations so that Mohammad does not have to re-calibrate the values.

The question we have is about the value range coming from the face capture system since it does not seem to be a simple -1 to 1 range from what we can see.

According to the original specifications we discussed, the gaze range goes between -1 to 1 for both horizontal and vertical, in which the the |1| values represent looking off into the distance (I.E. Thinking). When we test the system however, the range seems to be between -.3 to .6.

If we could just get the exact specification of what the face tracking system is returning we can make sure that any changes we make to the animation system will still work with it.

--Lazlo

— Reply to this email directly or view it on GitHubhttps://github.com/always-on/always/issues/269#issuecomment-34906531 .

Email: rich@wpi.edu Phone: 508-831-5945 Fax: 508-831-5776 Home: http://www.cs.wpi.edu/~rich

always-on / always

Agent jerky head movements in games #269

Sent from my Droid...