Decide how we round the scores

[lw]

At the moment the ScoreTypes are rounding the score and the max_score (both the public and the private ones) at two decimal places and, where applicable, the subtask score isn't rounded at all (we're using the %g output format).

At the IOI we were required by the rules to round the score to the nearest integer and patched our code to do so. Since we had many variable-score subtasks we also decided to round their score at two decimal places.

I think it's now time to decide what to do: keep the current rounding behavior, use the IOI one or provide some kind of configuration options for it.

[giomasce]

I'd say it should be configurable. It should also be configurable when to do the rounding: for each subtask, for each task, for the whole contest...

Anyway, definitely a low priority issue.

[lw]

It should also be configurable when to do the rounding: for each subtask, for each task, for the whole contest...

Specifying the contest-level rounding may not be that easy since the only point where this score is actually computed is RWS. Anyway, I think we just need to specify the rounding precision for each "level" (subtask, task, contest). Then we just sum the rounded values to get the score for the level above, round them, etc.

[stefano-maggiolo]

May I remind that the notion of "subtask" is only available inside the score type, whereas task and contest are concept of which the whole CMS is aware. I suspect that this fact may force us to split the configuration of the rounding into two parts: one (eventually) in the score type parameters, and one in cms.conf.

[lw]

Well, surely not in cms.conf. There's no need to put them there and to force to restart the services in case these values change. It's enough to put them on the Contest and Task objects. In the Task object the right place to put it is in score_parameters (BTW, why isn't it called score_type_parameters?). And coincidentally that's also where the subtask parameters are stored! There's no field like score_parameters for Contests because, as I said, the per-contest score is never computed in CMS (except in RWS).

[stefano-maggiolo]

Yes, you are right about Task/Contest and not cms.conf, I was sleeping :)

But what I was trying to say still holds: the rounding at contest and task level should not be in the score parameters because they do not depend on the specific score type.

This even if up to now there are no CMS services computing the total score (as there will be soon). On the other hand, subtask rounding must be inside score parameters as it is the only place where "subtask" makes sense.

[giomasce]

Il 12/10/2012 17:29, Luca Wehrstedt ha scritto:

It should also be configurable when to do the rounding: for each subtask, for each task, for the whole contest...

Specifying the contest-level rounding may not be that easy since the only point where this score is actually computed is RWS.

This must be fixed anyway: RWS must be just a showing tool, (official) scoring must be computed somewhere else. More generally, it should be possible to have a contest completely without RWS.

Giovanni.

Giovanni Mascellani mascellani@poisson.phc.unipi.it Pisa, Italy

Web: http://poisson.phc.unipi.it/~mascellani Jabber: g.mascellani@jabber.org / giovanni@elabor.homelinux.org

[giomasce]

BTW, in the meantime I pushed to master the commit 22d061d, which is how we did things at IOI.

[giomasce]

I try to sum up my views about how the score should be computed. I believe it to be the most easily usable and configurable mechanism that we can offer to our users.

• A TaskType evaluates each test case assigning it an outcome (that doesn't need to be a number: it can be everything; probably the best thing is to decide it to be a string and than JSON-encode in it what we need). Of course no rounding must be done at this level, since there doesn't even exist a concept of "rounding".
• A ScoreType implements some kind of function that mixes all the evaluations to a score. This score is in principle a floating point number and it is meaningful to round it to some number of decimal places. This should be configurable Task-wise (I'm not sure about which field is better entitled to store this piece of configuration, because, actually, I think the Task configuration fields deserve a bit of renaming).
• For each task, the tokened and the last submission are considered: thier supremum is considered the current score for this User on this Task. There is no need for rounding here, since the rounding is (believed to be) a monotonic operator, thus is commutes with lattice operations like max.
• The scores for all the Tasks are summed up to make the total current score for the User. I don't think it is necessary to offer other filtering functions different from sum. Anyway, sum doesn't even think about commuting with the rounding, so another step of rounding is necessary. It should functionally identical to the one at Task level, although configurable at Contest level.

The number coming out from the last step is the total current score and is compared with the same number calculated for other Users to determine the ranking.

Do you agree with this behavior?

[lw]

Yes. You're proposing two score rounding rules, one at task-level (used to round the score of a user on a task, i.e. the the score of a submission) and one at contest-level (to round the sum of the task scores). These apply to all tasks, and could perhaps be put in some dedicated fields on Task and Contest objects (e.g. "score_decimal_places").

But since some ScoreTypes may have other internal sub-scores (i.e. testcase score, subtask score, etc.) we should also provide a way yo specify the rounding of these sub-scores. The only way I see is to use the "score_parameters" field (which I propose to be renamed to "score_type_parameters", for consistency).

[giomasce]

Yes, you're right. Anyway, such additional rounding level is buried inside the ScoreType and is not visible by CMS. And, of course, their configuration is specific to the ScoreType.

[lw]

Partially fixed in c6ced820cb6744c773d3296ea3faa89c31b298f5. That is, rounding task- and contest-level scores works fine all across CMS (from CWS to AWS and RWS). The subtask/testcase/etc. rounding stuff is missing (the one that stays "inside" the ScoreType).

I chose to save the "full score" (without any rounding) in the DB and to round it only when displaying it. This allows to change the number of decimal places without having to rescore all submissions. On CWS and AWS the rounding is done server-side, while on RWS it's done client-side (this means that a user could potentially find out the unrounded score...).

I'm un-assigning me from this issue, thus everybody is welcome to step in and do the subtask/testcase/etc. part.

[stefano-maggiolo]

I would say we make sure that the default behavior for the subtasks is to implement the IOI rules (if allowed, keeping all the digits and leaving the rounding to the visualization) and we close the issue (or retarget for 1.x for more flexibility in the rounding inside the score type).

The only problem is that I don't remember how subtask were scored in the IOI :)

[lw]

At the IOI we were rounding the subtask scores to two decimal places, while the task- and contest-level scores were rounded to the nearest integer. (I already wrote that in the first message of this issue)

Do you mean that we should not implement "configurability" of score precision "inside" ScoreTypes for v1.0?

[stefano-maggiolo]

Yes, I'm all in favor of not changing more admin-visible stuff before 1.0. Since it now makes sense, we could remove the rounding to two decimal places in the score types.

[lw]

I don't really agree. Removing any rounding from the ScoreTypes code would mean that an user, through CWS, could see something like "Subtask 2 - 12.3456789 / 13" and this is ugly. (We could argue that this could give the user a way to retrieve information about input data, by tweaking it's output a little and seeing many bits of this unrounded number change). Thus, if we don't want to make subtask rounding configurable, we should set it to something reasonable, say two decimal places. But then it wouldn't make sense to set the score precision of tasks and contests to anything greater than 2. (Ok, probably many contest admins wouldn't have done it anyway...).

Thus, I think it's better to implement it. I realize that this will make the score_type_parameters "interface" even more complicated...

[stefano-maggiolo]

Oh, I see, I forgot that subtask scores are public (not worried about the hacking, but about the ugliness). Still, I don't like the idea of introducing new parameters into score_type_parameters before 1.0, so I'd keep the rounding at two decimal places.

[lw]

[...] so I'd keep the rounding at two decimal places.

We're already doing that. See the template in ScoreTypeGroup. (Sum doesn't have "testcase scores", so no rounding is needed there). Note that we're doing it only at a presentation level, that means that the global score of the submission will be the sum of the _un_rounded subtask scores.

round(1.234 + 1.234 + 1.234, 2) != round(1.234, 2) + round(1.234, 2) + round(1.234, 2)

[lw]

PS: what I wanted to say is that since we're already doing that we can close this bug or retarget it for v1.x (and this is exactly what I'll do).

[stefano-maggiolo]

Go ahead :)

[lw]

Little recap: to close this bug one needs to implement configurability of testcase/subtask score-rounding inside ScoreTypes using score_types_parameters.

[giomasce]

Well, first we should probably understand which "rounding model" we want to use. I made a proposal and don't understand whether it has implicitly accepter or not. I don't see any other proposal, tough.

In case my proposal is accepted, one should also implement a configurable rounding at contest level, after summing up the task scores.

[lw]

It's already there. Contest.score_precision controls exactly that.

As I said, the only part that's missing is the rounding of internal ScoreType elements (testcases and subtasks). This rounding will have effect on the data that's shown to users (cia CWS) and to admins (via AWS) and, probably, also on the score that's assigned to the submission (as it will be the sum of the rounded partial scores, and will therefore have the same rounding).

[giomasce]

Sorry, I missed it. So, I agree.