IOL segmenting objects but no perceived change in state

[dcam0050]

Hi everyone! Just a general intro, my name is Daniel Camilleri and I've just started a job as a research assistant with the WYSIWYD team in Sheffield. Any help with the problem below would be appreciated.

After successfully managing to segment the object using iol, I was trying the rest of the interaction to teach the iCub what the object is by saying "what is this?". Now, as the second screenshot shows, iol_main.lua is receiving an acknowledgement that the text has been received and recognized but I did not get any sort of response from the icub.

There must be a link of some sort missing or malfunctioning but I can't figure out which. Does anyone have an idea as to how I can debug this problem? Thanks!

[Tobias-Fischer]

Hi @dcam0050 and welcome to the WYSIWYD team,

I guess it is easier to ask the developers of iol (i.e. opening an issue there), as we guys at WYSIWYD don't use the lua scripts. We only use a part of the iol modules, and then use iol2opc to interface iol with our OPC. We have an XML file here in the repository which shows the modules and their connections for this setup.

Best, Tobias

[pattacini]

Hi @dcam0050

Are you running the complete iol demo or the subset of it designed specifically to interface object recognition to wysiwyd? It seems you've gone with the complete demo since you're talking to the iCub. Well, that's challenging since we don't have any documentation for that yet :smirk:

However, the what-is-it command needs to be executed along with a pointing action by the user who is required to indicate the desired object. If the pointing is not detected the iCub should utter a sentence warning the user about that.

Is the speech synthesis up and running? We make use of proprietary package for that (i.e. acapela) and it might be that iSpeak doesn't get configured properly simply because it doesn't find that package. Anyway, iSpeak can be run with festival too, which is free, but it still needs to be configured: that can be achieved by changing the relative options in the xml application you launch.

Once you did make sure that iSpeak is ok, try out the where is the toy? command instead. It's easier and does not necessitate any pointing at first. The iCub will reply he does not know the toy and then you'll have to point at it by producing some motion cue on the right location (wave your index finger on top for a couple of seconds) while saying "No, here it is".

At this moment, if everything else is in place, you should see the toy correctly recognized.

Let's start with these preliminary checks.

/cc @vtikha

[pattacini]

@Tobias-Fischer I've actually suggested @dcam0050 to post here because I thought he was using the wysiwyd layer and honestly I would also recommend to stick to it.

[dcam0050]

@pattacini I tried "where is the toy" and I've tried "what is this" while providing a motion cue but on both occasions I have not received a response from the icub. I also checked that iSpeak is functioning correctly by sending a string directly to the port and it works OK.

Could you tell me how the data flows from the speech recogniser and eventually to iSpeak? what ports and processes are involved in receiving and acting on the command received?

[dcam0050]

Update: I've traced through the code and seems that the problem lies within iol_interact_fsm.lua. This is the original code.

Which I've modified to this for testing purposes:

The modification results in this:

It seems like the variable cmd is not extracting the required section of the result string because print(cmd) returns nothing and print(event_table[cmd]) returns nil while print(event_table["See"]) returns e_exit as required.

@pattacini Do you have any ideas on how to correct local cmd = result:get(1):asString() because I'm quite new to lua? I'm using lua5.1.

[maxime-petit]

Yop,

For me, result:get(1) is a bottle ("where is the Toy" (rootRule{"#Object" Toy))), not a string. So the asString() does not work. You need to use toString() or get(1).asList().get(0).asString for extracting "Where is the Toy"

./Max

2015-10-14 13:34 GMT+01:00 Daniel Camilleri notifications@github.com:

Update: I've traced through the code and seems that the problem lies within iol_interact_fsm.lua. This is the original code.

[image: screenshot from 2015-10-14 13 00 01] https://cloud.githubusercontent.com/assets/4288535/10482877/afb589ee-7273-11e5-8852-73f3b9005aac.png

Which I've modified to this for testing purposes:

[image: luamod] https://cloud.githubusercontent.com/assets/4288535/10482876/adefd556-7273-11e5-9e75-bd54766ab83b.png

The modification results in this:

[image: screenshot from 2015-10-14 13 33 09] https://cloud.githubusercontent.com/assets/4288535/10483586/2661407a-7278-11e5-8a03-d83ec92a7e52.png

It seems like the variable cmd is not extracting the required section of the result string because print(cmd) returns nothing and print(event_table[cmd]) returns nil while print(event_table["See"]) returns e_exit as required.

@pattacini https://github.com/pattacini Do you have any ideas on how to correct local cmd = result:get(1):asString() because I'm quite new to lua?

— Reply to this email directly or view it on GitHub https://github.com/robotology/wysiwyd/issues/26#issuecomment-148036037.

[maxime-petit]

Hum...

Just a thought also, because the weird thing is I think this piece of code was supposed to work before no?

How do you launch the speechRecognizer? Do you use or not the --noLegacy parameters? Try to use it, and it might work.

This parameters change a bit how the bottle of the recognized sentence is built if I remember correctly so it may be it

./Max

2015-10-14 13:53 GMT+01:00 Maxime Petit maxpetit@gmail.com:

Yop,

For me, result:get(1) is a bottle ("where is the Toy" (rootRule{"#Object" Toy))), not a string. So the asString() does not work. You need to use toString() or get(1).asList().get(0).asString for extracting "Where is the Toy"

./Max

2015-10-14 13:34 GMT+01:00 Daniel Camilleri notifications@github.com:

Update: I've traced through the code and seems that the problem lies within iol_interact_fsm.lua. This is the original code.

[image: screenshot from 2015-10-14 13 00 01] https://cloud.githubusercontent.com/assets/4288535/10482877/afb589ee-7273-11e5-8852-73f3b9005aac.png

Which I've modified to this for testing purposes:

[image: luamod] https://cloud.githubusercontent.com/assets/4288535/10482876/adefd556-7273-11e5-9e75-bd54766ab83b.png

The modification results in this:

[image: screenshot from 2015-10-14 13 33 09] https://cloud.githubusercontent.com/assets/4288535/10483586/2661407a-7278-11e5-8a03-d83ec92a7e52.png

It seems like the variable cmd is not extracting the required section of the result string because print(cmd) returns nothing and print(event_table[cmd]) returns nil while print(event_table["See"]) returns e_exit as required.

@pattacini https://github.com/pattacini Do you have any ideas on how to correct local cmd = result:get(1):asString() because I'm quite new to lua?

— Reply to this email directly or view it on GitHub https://github.com/robotology/wysiwyd/issues/26#issuecomment-148036037.

[maxime-petit]

Replying through google seems to be a bad idea :warning:

Hum...

Just a thought also, because the weird thing is I think this piece of code was supposed to work before no?

How do you launch the speechRecognizer? Do you use or not the --noLegacy parameters? Try to use it (or remove it if it was used), and it might work.

This parameters change a bit how the bottle of the recognized sentence is built if I remember correctly so it may be it

./Max

[dcam0050]

I tried the --noLegacy parameter because it was not there before but I got the same response back. The modification you suggested extracts "where is the toy" but the code requires just the first word. How is this done in lua?

[pattacini]

@maxime-petit gave a good hint: that's element is a still a list, not yet a string. We do not employ any particular option to launch speechRecognizer indeed.

@dcam0050 iol is a very intricate bundle of software, hence it turns to be quite difficult to figure out what might be going wrong or even to try collecting meaningful data to explain you how it works underneath (this activity is somewhat time consuming for us now). That's right what happens when documentation is missing and we never find useful time slot to write it up :smirk: Our bad ...

That's why I think it'd be worth proposing a TeamViewer remote session to tackle this problem. Give us time to find a good moment for that: it could be the beginning of the next week, I presume.

/cc @vtikha

[vtikha]

@dcam0050 I have been following this issue and trying to figure it out with @pattacini but as he mentioned the best way to figure this out is accessing the system remotely (teamviewer) and figure things out hands on. If this is ok with you guys we can arrange a time slot and spend time on this asap. Please let us know a few dates and times when you will be available to do this in order to find the best slot that suits everyone.

[dcam0050]

@vtikha @pattacini I am available most of the time so let me know when it is ok for you and we'll arrange a team viewer meeting.

[pattacini]

Unfortunately we could be available only starting on Monday 19. Is that ok for you?

[dcam0050]

Monday 19th is good. At what time?

[pattacini]

I'd say 10ish.

[vtikha]

@dcam0050 We are ready to go :+1: . Please reply to Ugo's email with the details for the connection.

[vtikha]

@dcam0050 @Tobias-Fischer @maxime-petit (just for keeping track of things).

During the remote debugging with @pattacini we noticed that:

• The speechRecognizer was automatically launched with the --noLegacy param from the installed config file and therefore the output was completely different. Please make sure that you launch it with the correct parameters when IOL is required (or you might even want to create a new config.ini for IOL just not to mess up your other experiments).
• The himrep and the database where not synchronized, and was trying to work with a dictionary that was not available. By cleaning up the files and started from scratch everything managed to connect to each other.

After these were resolved, it all worked like a charm. Thanks @dcam0050 for bearing with us and doing all the tests from your end.

[Tobias-Fischer]

Hi @vtikha and @pattacini , great to hear you were able to resolve this issue! Just out of interest: which parameters should the speechRecognizer be launched with?

Thanks, Tobi

[pattacini]

Hi @Tobias-Fischer

Our config file looks like the following:

grammarFile    smallGrammar.grxml
timeout        20000

[Tobias-Fischer]

Great, thanks!

[maxime-petit]

Yop,

Ok so no --forwardSound (but should have no influence except if you want to record it into the ABM) and no --noLegacy for the iol (the "classic" one, not the iol2opc from WYSIWYD) But be careful, if you want to use the speech with WYSIWYD speech-based module, you will need the --noLegacy option.

./Max

[vtikha]

@maxime-petit Exactly :+1:

[dcam0050]

Thank you @vtikha and @pattacini very much for your help with this problem!