[info-terminal] Create deployable system.

[cburbridge]

These are the info-terminal team members

• Akshaya Thippur @akshayats (KTH)
• Bruno Lacerda @bfalacerda (BHAM)
• Lenka Mudrova @mudrole1 (BHAM)
• Eris Chinellato @eris-ch (Leeds)
• Tom Krajnik @gestom (Lincoln)
• Umer Rafi @urafi (RWTH)
• Michael Zillich @mzillich (TUW)
• Chris Burbridge @cburbridge (BHAM)

There are still quite a few things to be done in order to be able to deploy the info-terminal:

• [ ] Add install targets for packages
• [ ] Tidy up package.xml files
• [ ] Fix the PTU behaviour. When the robot is driving, PTU forward. When info-terminaling, backwards.
• [ ] Check/Fix the timeout: At AAF the timeout period did not appear to be always working.
• [ ] Make sure that once a Drive to person and info-terminate them task has been demanded, no new card detection triggers another one until that person has done with the robot.
• [ ] Add a Go Away button to the screen for person-called info-terminal. Also active during scheduled info-terminal sessions, maybe allows the robot to learn not to schedule info-terminal task at certain nodes.
• [ ] Replace the circle/card detector with a upperboddy & gesture detector.
• [ ] Check that the person is still there when driving to them.
• [ ] Improve the weather display so that it shows today & tomorrow, with a little more detail and slightly smaller/better resolution weather images.
• [ ] Create picture slideshow screensaver like screen for while robot moves between info-terminal points.
• [ ] Create an events entry interface for staff, remove the BBC RSS feed from the events page.
• [ ] Create a cafeteria menu entry interface for staff
• [ ] Create overall admin interface linking to the three sub-interfaces.
• [ ] Include check for date/time before showing food menu to make sure that the correct menu is shown.
• [ ] Use camera to detect nobody present in addition to screen touches for timeout.
• [ ] Verify that the robot eyes are controlled in a desirable way...
• [ ] Switch to the "person-approacher" action in TomK's info-terminal scheduler.
• [ ] Consider renaming "person-approacher" to reflect that it does not have to always approach the person.
• [ ] Further test TomK's task demander.
• [ ] Write documentation
• [ ] Create launch files
• [ ] Test info-terminal scheduling within a larger schedule/routine.

Most of these tasks are quite minor fixes/improvements to the functionality that we developed during the winter school, but it worries me that we never got a draft of an upper body/ gesture detection to replace the circle detector. @mzillich do you know the state of this? Is there any code on github from the sub-group working on this? If this is not going to work then we will have to fall back to pre-scheduled info-terminal waypoints only.

@bfalacerda @ToMadoRe @gestom @akshayats @mudrole1 Please check the list, add to it anything I missed.

[mzillich]

Umer had a prototype working, still in matlab. It worked ok for the examples he tested. I will ping him on progress.

[mzillich]

Thanks for the list! Can we attach names to these tasks?

[marc-hanheide]

Quite a number of these things on the list are related to #20 so I feel the info_terminal group should be looking into the staff interface as well.

For the gesture detection, we need a working prototype ASAP as I feel this is a critical functionality and we need to assess its reliability to be able to decide upon its inclusion in the deployed system.

[mzillich]

@urafi How is the stop gesture recognition coming along? When do you think we can have a ROS component for that? Does not have to be perfect at this point, let's take what we have and see how well it integrates into the scenario.

[marc-hanheide]

I suggest to put it into the activity_recognition repo and open a dedicated issue there to discuss its integration.

[mzillich]

I will put it in strands_perception_people, where also the upper body detector resides.

[marc-hanheide]

OK, fine by me, let the repo maintainer @cdondrup have his say on this.

[cdondrup]

Since the head orientation is there as well, I think we can put it there. I will add some remarks in the issue.

[denisehe]

Hey,

so how is it planned: it is you who are providing the interface layouts and we provide you with pictures, music etc. that you then fill in bevore the deployment or is it that you provide the interface and layout and we can add content?

[denisehe]

@replacing card symbol to start the infoterminal: as michi and I discussed during the winterschool, maybe looking and waving at the robot (both taken together) would make a natural as well as a distinct sign for the robot to start displaying the info terminal.

[cburbridge]

There are 3 elements of the info-terminal that you will be able to edit during the deployment:

• the cafeteria menu
• the current events
• the photo album

The layout of each page will be fixed, with just the content being editable. Editing/adding content will be from a web page that you can view on your desktop machine. The photo album editor (implemented already) allows you to upload new photos in JPG format, and delete existing photos. The current events manager will allow you to upload a photo and some text, and delete old events. The menu will be editable in the same way, but will take a full weeks menu (only the next meal is shown to the user).

[denisehe]

okay! that sounds great!

and as we also were talking about displaying news informations on henry:

would it be possible to get a link to www.orf.at - one of austrias main news pages.

Could we restrict the internet function to just that side, so that users can not surfe somewhere else? Could you test if Henry's display shows that side properly and if it is "easy" to use via the touchscreen or if it is to small and rather difficult to klick headlines and navigate through the webpage?

[mzillich]

Umer is working on the waving gesture. So the plan is to replace the cards.

[denisehe]

(Y) cool!

[denisehe]

I sent out one of our lunch menues but via email as github did not accept a pdf attachment

[marc-hanheide]

Here's a link to the menu

[mzillich]

Umer's simple stop gesture detector developed at the school seems not fit for deployment. He is working on the full proper skeleton based one. But we have yet to get a time line for that.

[marc-hanheide]

Also, from limited experience, a skeleton one will struggle with several people. @urafi we need to be aware of the requirements/challenges:

• several people in the field of view
• a moving robot/camera (i.e. motion blur and non-static background)
• people not in "star-fish" shape, i.e. initialisation as star-fish is not required I understand, but standing in "random" pose when we encounter humans will be happening.
• occlusion by people walking past
• limitation of computational resources (we can't dedicate more than one core to this, I'd say)

Please, @urafi, comment if you deem there to be a feasible implementation ready to be included in a months time. Otherwise, I suggest to postpone the integration of this feature and rather go for something simpler, based on an analysis of a person's trajectory and "facing the robot" time or likewise.

[denisehe]

hey :-)

i just wanted to feed back that the infoterminal GUI looks great! really cool :-) i showed it to my colleague and she also considers it very nice, clear and stylish! :-)

cheers Denise

[mzillich]

Hi @urafi, any news on the stop/waving gesture detector? Do you think an implementation will be available before the code freeze? Or do we have to rely on alternative? Thanks, cheers

[marc-hanheide]

I think, in the last AAF hangout, we agreed that it is not fit for deployment and decided not to include it. But @urafi should still respond on this ASAP. The info-terminal task has been amended in the following ways (right, @yianni?):

• Effectively, info-terminal shall be active as the "default" task, i.e. active whenever the robot is "just patrolling".
• Given that, people shall be enabled to stop the robot by touching the screen "anywhere" (intercepting the onclick on the whole html body) and use the info-terminal. (see #29)
• We will defined a number of "info-terminal enabled" waypoints where the robot will "rest" and wait for people to use the info-terminal. If nobody uses it, the robot will continue after a set time with its patrol.
• The waiting time shall be determined by Fremen, i.e. waypoints where people actually used the info terminal will incur a longer rest time, as it is believed that people "liked" the feature at a specific place and time (@gestom to comment on this).

The schedule suggested by @denisehe looks like this: I suggest that "Lobby" is translated into all the suitable waypoints in the lobby, and so is "Ambulance" where we shall have Henry roaming between those waypoints. In term of task, this means that info-terminal has to provide a "patroller task" that can be configured with waypoint sets that it patrols actively, and adjusts the waiting time according to the fremen experience. Please @mzillich, comment on this and assign people to the required sub-tasks. I'm happy to discuss in person on Thursday when I'll be in Vienna.

[marc-hanheide]

sorry, I didn't mean @yianni (he's bellbot after all), but I meant @cburbridge...

[hawesie]

@cburbridge is not the leader of that task, he's pretty much up to his eyeballs with security.

[marc-hanheide]

I know he isn't, but @mzillich is, and he missed the meeting we agreed this, so I wanted someone else to confirm what we agreed upon. I'm not trying to sneakily steal him... (or, I won't admit it).

[cburbridge]

I am pretty sure that is what we discussed: drop the gesture recognition to make sure we have a robust system that does not inadvertently approach people who dont want to see it.

[marc-hanheide]

@yianni can you poke @urafi. I'm not sure he reads his github notifications...

[mzillich]

@gestom wrote:

the FreMEn is so far tailored to model probabilities of binary states, e.g. the probability that someone will interact with the robot at a particular place if the robot waits for a certain amount of time. Therefore, it will simply cause the robot to visit more populated places more frequently.

Depending on the robot's strategy (e.g. taking into account the extra energy spent while moving), it can prefer to wait at a given location instead of going somewhere else.

We might also represent the chance that an interaction will happen within 'n' minutes as 'n' different actions, where n \in {1,2,3,4,5}. A robot arriving at a particular position would then choose an 'optimal' (again depending on its strategy) time to wait.

[mzillich]

@gestom : is the above already implemented, and running in the system?

[gestom]

Apart from the 'We might ... time to wait' part, it is implemented and was running when we have demonstrated the infoterminal. As far as I know, the robot now waits for one minute and assumes that it can get to a given spot in 2 minutes. The robot uses a Monte-Carlo-based method to choose a location to go and his preferences (weights for Monte Carlo) are calculated as w_j = a_H_j(t)+(1-a)_p_j(t), (1) where p(t) is the probability of the interaction at location j and time t. The H_j(t) quantifies how much a visit to the location j will improve the FreMEn-based probabilistic model that constitutes p(t). Thus, the constant 'a' in (1) defines an exploration/exploitation ratio. Higher 'a' will cause the robot to prefer locations, where the probability of interation is about 0.5, leading to good models of p(t). Lower 'a' will cause the robot to visit places, where it is likely to be interacted with, but the obtained probabilstic models of the interacton will not be that good.

[gestom]

Just a comment on the fixed schedule that @denisehe suggested:

The robot is supposed to learn such a schedule on itself, that is one of the (scientific) aims of the STRANDS project. If we fix the schedule, how can the robot improve its behaviour over time ?

IMHO, the system should be doing infoterminal by default and bellbot and walking groups on demand. It should LEARN FROM EXPERIENCE that the bellbot and walking tasks are scheduled on a regular basis and do info-terminal near the areas where the bellbot and walking groups start, so that it can reach these promptly when requested. Also it should LEARN that that sometimes, there is a better chance of getting interactions at the ambulance rather than at the lobby. This is what the project promised to demonstrate.

[mudrole1]

@mzillich According your email... the timetable posted by Marc can be written down to the robot's routine. I am happy to write down such a routine, but each task needs to have its action defined (by the team responsible for it.) I am not going to write the action myself. However, the writting down the routine was kind of easy solution last year and it much more fits security scenario. As pointed out by @gestom , we should aim not fixing the routine if possible.

[marc-hanheide]

@mudrole1 I think there is no need to write the routine, as we will be using The Google Calendar Interface I made: https://github.com/strands-project/strands_executive/pull/132 And, that's what we need priorities for, we will have a "fremen-ised" low priority routine that suggests info-terminal tasks all the time (e.g. when not overwritten by some higher-level tasks)