leaderboard column names

[ivaniadg]

I keep getting this error when submitting a score to the leaderboard.

Traceback (most recent call last):
  File "./apps/web/views.py", line 492, in get_context_data
    context['groups'] = phase.scores()
  File "./apps/web/models.py", line 715, in scores
    'resultLocation': result_location[count]
IndexError: list index out of range

My scores.txt file has this

nDCG: 0.0 MAP: 0.0 RMSE: 0.0

and the competition.yaml file has this

leaderboard:
    leaderboards:
        Ranking: &Ranking
            label: Ranking
            rank: 1
        Rating: &Rating
            label: Rating
            rank: 2
    columns:
        nDCG:
            leaderboard: *Ranking
            label: nDCG
            numeric_format: 3
            rank: 1
        MAP:
            leaderboard: *Ranking
            label: MAP
            numeric_format: 3
            rank: 2
        RMSE:
            leaderboard: *Rating
            label: RMSE
            numeric_format: 3
            rank: 1

What am I doing wrong?

[ckcollab]

Are you trying to do multiple leaderboards? Rating vs Ranking? I believe you can only do one, I would try just doing Ranking? If possible?

[ivaniadg]

I thought we could do it. I saw it in the competition "MICCAI Multimodal Brain Tumor Segmentation (BRaTS) Challenge". They had two tables with all the results.

I used their competition as an example to create my competition. Their code is here

I'm able to show both tables in the results tab but once I send results to the leaderboard it crashes

[FlavioAlexander]

@indonoso Were you able to find a solution for your problem?, If so, I will close this issue

[ivaniadg]

Nopes I didn't :( I just put everything in one table. I think it would be useful to see the code they used to populate those tables.

[bethard]

I'm getting the same error. The display code seems happy to allow multiple leaderboards, and I'm able to get a display similar to that of the BRaTS Challenge:

However, after a single "Submit to Leaderboard", I get:

The error is from this part of the code:

https://github.com/codalab/codalab-competitions/blob/e0b8f8ed43d7037e91b1d2439d0bb9c189978e79/codalab/apps/web/models.py#L847

The problem seems to be that the list returned by:

SubmissionResultGroup.objects.filter(phases__in=[self]).order_by('ordering')

is larger than the list returned by what's used to create the result_location list:

PhaseLeaderBoardEntry.objects.filter(board=lb)

Any idea why that might happen?

[bethard]

Digging in a little further, I see that resultLocation is only used if if phase.competition.enable_medical_image_viewer, which makes me suspect this is a bug that was introduced with the code for enabling the medical image viewer.

It seems like there's something sketchy in the code that tries to keep result_location and submission as parallel lists. Rather than doing that, it seems like it would be simpler to just have that submissions list store triples instead of pairs. Here's a proposed patch:

diff --git a/codalab/apps/web/models.py b/codalab/apps/web/models.py
index 90beba1..fdc5f14 100644
--- a/codalab/apps/web/models.py
+++ b/codalab/apps/web/models.py
@@ -811,7 +811,6 @@ class CompetitionPhase(models.Model):

         # Get the list of submissions in this leaderboard
         submissions = []
-        result_location = []
         lb, created = PhaseLeaderBoard.objects.get_or_create(phase=self)
         if not created:
             if include_scores_not_on_leaderboard:
@@ -820,15 +819,11 @@ class CompetitionPhase(models.Model):
                     status__codename=CompetitionSubmissionStatus.FINISHED
                 )
                 for submission in qs:
-                    result_location.append(submission.file.name)
-                    submissions.append((submission.pk, submission.participant.user))
+                    submissions.append((submission.pk, submission.participant.user, submission.file.name))
             else:
                 qs = PhaseLeaderBoardEntry.objects.filter(board=lb)
                 for entry in qs:
-                    result_location.append(entry.result.file.name)
-
-                for entry in qs:
-                    submissions.append((entry.result.id, entry.result.participant.user))
+                    submissions.append((entry.result.id, entry.result.participant.user, entry.result.file.name))

         results = []
         for count, g in enumerate(SubmissionResultGroup.objects.filter(phases__in=[self]).order_by('ordering')):
@@ -837,14 +832,14 @@ class CompetitionPhase(models.Model):
             scores = {}

             # add the location of the results on the blob storage to the scores
-            for (pk, user) in submissions:
+            for (pk, user, file_name) in submissions:
                 scores[pk] = {
                     'username': user.username,
                     'user_pk': user.pk,
                     'team_name': user.team_name,
                     'id': pk,
                     'values': [],
-                    'resultLocation': result_location[count]
+                    'resultLocation': file_name
                 }

             scoreDefs = []
@@ -891,7 +886,7 @@ class CompetitionPhase(models.Model):

         if len(submissions) > 0:
             # Figure out which submission scores we need to read from the database.
-            submission_ids = [id for (id,name) in submissions]
+            submission_ids = [id for (id,name,file_name) in submissions]
             # not_computed_scoredefs: map (scoredef.id, scoredef) to keep track of non-computed scoredefs
             not_computed_scoredefs = {}
             computed_scoredef_ids = []

I don't know how to test this, but I feel reasonably confident that the fix is something in this direction. Is there anyone on the project that can try this out?

[FlavioAlexander]

@bethard I will need some time to dig into this issue. It seems like it was possible to have multiple Leaderboards before, not sure if your competition is failing because of a bug with the platform or some other error such as score script.

[bethard]

I can confirm its not my scoring script. If I use exactly the same scoring script, but simply remove the extra leaderboards from competition.yaml (using the same single leaderboard for everything) the error goes away, and scores are displayed properly.

[bethard]

To help you dig into the issue, here is a minimal example. Make a program.zip with two files, metadata, which looks like:

command: python $program/ab.py $input $output
description: Prints a fixed set of scores

and ab.py which looks like:

import os.path
import sys
[_, input_dir, output_dir] = sys.argv
with open(os.path.join(output_dir, "scores.txt"), "w") as scores_file:
    scores_file.write("A: 1.0\n")
    scores_file.write("B: 0.5\n")

Then create a competition.zip file which contains the program.zip that you just made, as well as an empty.html file with no contents at all, and a competition.yaml file with the following contents:

title: Test multiple leaderboards
html: 
    overview: empty.html
    evaluation: empty.html
    terms: empty.html
    data: empty.html
phases:
    1:
        phasenumber: 1
        label: Main
        start_date: 2016-07-01
        scoring_program: program.zip
        reference_data: program.zip
leaderboard:
    leaderboards:
        A: &A
            label: A
            rank: 1
        B: &B
            label: B
            rank: 2
    columns:
        A:
            leaderboard: *A
            label: A
            rank: 1
        B:
            leaderboard: *B
            label: B
            rank: 2

Create a new CodaLab competition using that competition.zip file, make a submission (any .zip file should work), and visit the leaderboard. You should see the error that we've been seeing above.

In contrast, if you do exactly the same thing, but instead use only a single leaderboard in the competition.yaml, then the error will go away. If you want to try that, here is a competition.yaml that will work:

    title: Test multiple leaderboards
    html: 
        overview: empty.html
        evaluation: empty.html
        terms: empty.html
        data: empty.html
    phases:
        1:
            phasenumber: 1
            label: Main
            start_date: 2016-07-01
            scoring_program: program.zip
            reference_data: program.zip
    leaderboard:
        leaderboards:
            A: &A
                label: A
                rank: 1
        columns:
            A:
                leaderboard: *A
                label: A
                rank: 1
            B:
                leaderboard: *A
                label: B
                rank: 2

Let me know if there's anything else I can do to help.

[bethard]

Any news? This is blocking us from using codalab for SemEval.

[Garrafao]

Hi, it seems that the issue was closed without a solution. I have a similar problem. Is there any way to use multiple leaderboards? Thanks in advance!