Closed leonardoSCosta closed 1 year ago
@tobiasheineken I don't really see you're point. Yeah the vision is better calibrated during the finals, but that doesn't effect the group stage.
During a game both enemys have to deal with the same issues and thoughts like "If we had better vision during the group phase we might have won the game" aren't really sportsman like, are they?
@andre-ryll already mentioned it. It's a shared system and everyone has to deal with the issues. I personally think it's even fairer to have a consistent vision during a game. So no team can alter the vision during the game for their advantage.
You've accepted it during preparation, deal with it.
@mickmack1213 To make it absolutely clear: This is not a fairness issue. If the vision is not modified during a match AND both teams get enough time where all officials are already in place and where they can check the calibration and start a complaint, the game is fair. Even if there are slight changes, if they are done in good faith teams have to deal with it (see ER-Force vs. RoboteamTwente 2019, where Nicolai accidentally removed our ability to see the ball during half time)
However, fairness is not the only issue the rules have to cover. Having bad vision has bad effects on the game:
I think we all can agree that these points are bad - or at least some of these issues are.
The final vs. Group stage point i wrote earlier is prove two things:
Technically, all of these points are irrelevant as we have an vision expert who should be interrupting the game on its own. The vision expert interrupts the game, the calibration is improved, no team interaction is needed. At least that's the way the old rules where supposed to work.
In reality, it wasn't the Vision Expert, it was one of the teams. The reason for that is well known: SSL-Officials aren't always that good. If teams start a complaint it is no longer a "objective" thing, but it could be because teams are better or worse at handling small hiccups. Which is obviously a bad thing.
This pull request offers a new way into this problem: If we can find a good "minimum quality", and interrupt the game as soon the quality gets worse that it, we replace the vision expert's opinion by a measurable metric. Even better, teams now are able to know what to expect, they what their software should be able to handle. As the game will not be stopped inside of these parameters, teams have to work on the "deal with it" mentality that Andre mentioned.
The interesting part is: What is a good minimal quality? If the bar is too high, we interrupt the game all the time. If the bar is too low, all of the issues stated earlier in this comment start to arise. Also, we start to get towards a territory where the initial calibration is not improved during the tournament because "it was good enough".
I think that my numbers are a good compromise. They are still very bad and even good teams will struggle if the data is so sparsely. But at least they can test it and these numbers seem to be physically possible to work with. And Nicolais data suggests that they could (maybe should) be more strict in following years, but we could start with these.
However, Nicolai is right when he wrote that time between detection is not sufficient. There are still other issues (like extreme changes in orientation for some 7/8, changing id's for the other robots), that have to be covered. Also, I think that Andre's "detections per second" should also play a role here. We implemented this very low vision quality in our simulator and I can tell you that the ERForce autoref - which already can tolerate vision hiccups - will totally stop working if we continuously add information that sparsely (100ms between each detection for each robot).
So I suggest 0.2s for robots, 0.1s for the ball, and at least 20 detections per second. These values allow for a coordinated match, they are not overly perfect and encourage "deal with it", and they make sure ssl-vision is used to its full potential during robocup.
@tobiasheineken I appreciate your feedback but your posts tend to be rather long. Please try to condense them a little.
Of course a bad vision has some impact on the game but I wouldn't paint it as black as you did. If you have a specified minimum vision quality you can adjust your path planning accordingly to have larger safety areas to not crash into opponents. Of course this may set you back in case you have a more fearless oppononent. That's what sport is also about, planning rewards versus risks. Apart from that we have the second vision blackout challenge this year, encouraging teams to test with onboard vision. With that you have extra information for your risk planning. I see no technical difficulties here which cannot be solved.
The primary point remaining of this issue is how to define the quality. I would say 0.2s of blackout for robots sounds fair. The ball is a difficult matter. It naturally vanishes in robot shadows. We could only measure its quality if it is visible and all robots have some minimum distance to it (say, 0.5m). That imposes additional problems. If the ball vanishes completely because of very bad vision we cannot even judge if it is in a robot shadow or just not detected. Any idea on how we handle the ball would be great!
@andre-ryll
Apart from that we have the second vision blackout challenge this year, encouraging teams to test with onboard vision
While I agree onboard vision it is absolutely one way to help solve the problem and is worth investigating (especially for the ball disappearing in robot shadows), I don't think it should be used as a primary reason to accept significantly lower SSL Vision quality.
My understanding is that the SSL is set up the way it is with cameras and SSL Vision to allow teams to focus more on strategy and multi-agent coordination, not on perception, because this is the SSL's contribution to the 2050 goal of RoboCup to beat the Fifa champions. Allowing vision quality to be "too low" puts more emphasis on onboard vision or other perception solutions, and detracts from the main issues the SSL is trying to solve / research. Perhaps this is worth bringing up in the post-RoboCup meeting this year since I believe we will be discussion the "purpose" of the league at that time.
I also agree that 0.2s for robot blackout is a good number.
@andre-ryll There are two remaining points in this issue
Otherwise, I agree with your comment. Simply stating "unless the ball is covered in a shadow" is easy when writing the rules, but hard to very hard when actually enforcing it. If we assume that a ball cannot teleport, we should be able to check the blackout by using some assumptions:
I suggest to postpone this rule change to next year.
We can use the time to work on tools to supervise the quality and try to come up with reasonable numbers, instead of guessing numbers now that can not be checked by any present software anyway.
While I'm always in favor of "not changing too many things at once", I think postponing this is a mistake. Vision-Issues are a major part of delays and if we want to reduce them, we have to work here.
How about not changing anything in the rules themselves, but add a software support to detect any vision dropouts of 0.2s for any robot, while also adding a note that teams should be able to handle 0.2s 20 fps robots? In an ideal world maybe trigger ssl-vision to record images whenever this invariant is broken to fix the calibration post-match (which is one of the reasons I personally complain during matches, to fix it for future situations) ?
Isn't this what I said? Do not change the rules, but work on the software.
I've also started with something like you suggested. You are welcome to contribute or implement your own tool: https://github.com/RoboCup-SSL/ssl-quality-inspector
From the Rule proposal:
This change applies to both divisions