search

RoboCupAtHome / RuleBook

Rulebook for RoboCup @Home 2024
https://robocupathome.github.io/RuleBook/
Other
149 stars 61 forks source link

Automatic refbox where possible #80

Closed LoyVanBeek closed 6 years ago

LoyVanBeek commented 9 years ago

Some things are very hard to grade for the referees. For example:

kyordhel commented 9 years ago

+1 to refbox

Some robots (like amigo & Golem & justina) does not have a screen to be read. Also I don't like this annoying issue of teams blaming wifi because the robot was unable to notify. Neither I like cheating teams sending fake detection messages. This is also hard to coordinate.

I would prefer to add points to SRAD test or even to Robot Inspection for a loud and clear voice (teams should be forced to add nice on-robot speakers).

On the refbox.

LoyVanBeek commented 9 years ago

The scoring webapp would alsready be nice. It can just be a matrix of checkboxes for starters. I know that our @rayman can easily do this :smile:, right?

+1 for checking the voice to be absolutely clear. Can be tested in combination with the continue rule.

semeyerz commented 9 years ago

Maybe we should test the wireless audio transmitter for the speakers at the arena as well and find/mark good volume settings for all the robots. This could improve the speech quality a lot. Especially for the audience. And this audio stream could then easily be recorded/subtitled as well.

LoyVanBeek commented 9 years ago

@semeyerz this would be something to test in the robot inspection and registration.

kyordhel commented 9 years ago

I'm working on an app (sorry, is C# but works fine in ubuntu with mono) which listen an UDP port (no handshake) for incomming XML messages and logs them with the arrival time. The aim is robots send status messages to the app with the following information:

text to be synthetized as speech and maybe

recognized spoken commandsrecognized objectsrecognized gesturesrecognized activitiesother events

So the app can show the remaining test time and what the heck is happening on the screen, something like:

Robot said: "The one who lives in a pineapple under the sea is Sponge Bob Square Pants!"Robot said: "I can see 5 people in the crowd, three girls, one boy and Chuck Norris" Robot detected the human waking up!Robot saw a pot of weed

Plus, the app broadcasts 3 signals:

START signal when the test and timer start.TIME when the test starts and every minute after it, indicating the remianing test time.STOP signal when the test and timer stop. I will forward you the github repo once it is on a decent state.

Prost! Mauricio

LoyVanBeek commented 9 years ago

Sounds cool!

kyordhel commented 9 years ago

@LoyVanBeek

https://github.com/kyordhel/refbox.git

The current branch is server_GUI

LoyVanBeek commented 7 years ago

This also help to run your robot in complete Continuous Integration simulation

LoyVanBeek commented 7 years ago

Can at least the visualization part of a refbox be handled by a webpage served by the robot?

The robot would then serve:

We simply open these pages on a TV screen in the arena and the form on a laptop/phone for referee use only

balkce commented 7 years ago

It's worth mentioning that a trustee has emphatically voted against the use of a refbox. Apparently there has been a LOT of issues with it in other competitions.

iocchi commented 7 years ago

Let me emphasize some problems with wireless refbox: 1) Organizers must guarantee wireless network availability 2) When robots are not doing what expected and teams report that it is a network issue, it is difficult to understand what was the problem and how to proceed (wait until the network problem is fixed???) 3) Writing the code for a first version of a refbox is easy, maintaining it over the years is very difficult.

From my experience in other competitions, it is a strong commitment and a lot of work for the OC and at the end the return is very little. I would put instead more effort in improving the rules, making them easier to evaluate, so that refbox is not needed.

Alternative suggestion: Instead of a refbox developed by TC/OC, we can think of a nice UI developed by teams that is rewarded in the competition. TC/OC may write some guidelines about the functionalities of the UI and leave the teams to develop it. Using this UI will not be mandatory, but it will contribute with score (it can be one of the basic abilities for some tests). After a few years the best UI will be open source and all the teams will be able to use it and develop more.

LoyVanBeek commented 7 years ago

The refbox as suggested yesterday in https://github.com/RoboCupAtHome/RuleBook/issues/80#issuecomment-320444240 would run on the robot and not do actual refereeing, only visualization (maybe rename this bit to 'vizbox'?). Before opening the door, we require the team to open it's webpage on the big screen. If we know the WiFi is borked, too bad, no visualization for the audience.

As TC, we can set up a basic version, to be used by the teams and we accept pull requests by teams improving it.

kyordhel commented 7 years ago

If Memory fails me not, we set a guideline indicating that all suggestions must come as pull requests.

I would say the same is applicable to all monitoring and refereeing systems. I would like to see the beta running before even considering it, unless the federation provides funds for its development and long-term support. Then I would discuss the project.

iocchi commented 7 years ago

Great. A vizbox developed by teams would be perfect and overcome the problems I was mentioning above.

Please notice also that the RoboCup Federation provides funds for Projects for League Development that can cover this kind of activities. A call is usually issued in September/October.

LoyVanBeek commented 7 years ago

Some initial version provided by the league to the teams would help, to get some sort of template and basics in place. This could indeed be developed by such a 'Project for League Development'. How do we set that up?

LoyVanBeek commented 7 years ago

Oh, and of course the 'vizbox' could/should also display some info about the challenge/commands the robot is executing. Have the robot comment and narrate on what it's doing.

kyordhel commented 7 years ago

As long as, unlike with the PDF reports, we don't need to deal with all the "intuitive" designs of the teams, and lots of useless data (because more is always better), I'm OK.

An important point to remark is that guidelines will be necessary for the explanations provided by the robot. Robots must provide sufficient amount of information to allow the audience to understand what is happening, but without polluting the environment with nonsense speech, beeps, bleeps, and other high-speech sounds. Also it shouldn't be allowed to deceive the audience or provide obvious explanations (I can't grasp the object because I don't have a manipulator).

LoyVanBeek commented 7 years ago

Another reason to have this be a package provided to the teams. Initially developed by us, improved by the community.

moriarty commented 7 years ago

Running a VizBox on each robot, would then mean the OC or some volunteer accesses the webpage (ip address) of each robot, each being slightly different which means audience would see several different screens and the UX wouldn't be unified, which is confusing.

VS: running a unified VizBox developed by the league & teams, central one subnet above the VLANs, with a simple API: teams can request info, and post info to the VizBox., use simple token authentication if needed. It should be very simple, but allow for future expansion (when two teams are in the arena how should it behave). The goal would be turn it on once and leave it on for the entire competition. If V1, displayed the team name, the recognised speech, and the output audio as text, that would be better than nothing. Images are always a pain... they were in RoCKIn & @Work: usually because the transport layer and serialisation protocol need to be defined.

Either case, the wifi should be expected to fail and have no harm to the system, just lack a visualisation. This is easier to maintain running a centralised VizBox which we ask teams to sent some small heartbeat to, then we additionally are able to maintain an approximate list of who is connected to the wifi.

kyordhel commented 7 years ago

No web pages or any kind ok web-based interface. Apache uses many resources that must be better spent in solving the task. If there is any visualisation tool, it should only consume data from the robot, running all post processing in an external environment.

We don't want robots performance to decrease or even have crashes because the vizbox is tightly coupled.

moriarty commented 7 years ago

Nginx? You don't need Apache...

All the teams need is a small ROS proxy node (yes- we make the thing VizBox ROS argnostic, but face it, we know huge majority run ROS) which is event based: on receiving a string, it publishes the string to VizBox via http, if it can't, it fails quietly. Don't use service request model, just topics so that we don't care if it fails. It would need one or more request topics, where if you receive an request, make the corresponding API request and publish the result out on another topic. Loosely coupled, purely event based.

To use it: teams could simply launch it and remap existing topics to the proxy. The overhead wouldn't be much... if they're running Ubuntu desktop, then I'm sure there's worse culprits for processes degrading performance.

kyordhel commented 7 years ago

That solution sounds pretty straightforward. I liked it.

Are you volunteering yourself for the VizBox Alexander?

LoyVanBeek commented 7 years ago

@moriarty and I were discussing it yesterday over WhatsApp, so I started to whip something up: https://github.com/LoyVanBeek/vizbox Forgive me my drawing skills :-)

moriarty commented 7 years ago

Are you volunteering yourself for the VizBox Alexander?

Yes.

having worked on the RefBox/Central Factory Hub for other leagues, I'm glad this is a VizBox đź‘Ť

kyordhel commented 6 years ago

@moriarty Will we continuing with this now we have VizBox or shall we close this issue?

LoyVanBeek commented 6 years ago

VizBox is only visualization, no scoring at all. It's a start, lets see how it works this year and perhaps expand on that

moriarty commented 6 years ago

Yes, sorry I agree with @LoyVanBeek. When I checked out the VizBox I then didn’t spend any time on this.