"Submission Failed" on codabench.org without any additional information when checkbox is used

[johann-petrak]

I tried to test a codabench project on codabench.org and had a situation where I get the error pop up "Submission Failed" and nothing else whatsoever anywhere for every single submission. How is it possible to debug such a situation?

[johann-petrak]

This seems to be related to the following entry in the fact sheet configuration:

  "confirm": {
     "key": "confirm",
     "type": "checkbox",   
     "title": "Please confirm ...",
     "selection": "false",
     "is_required": "true",
     "is_on_leaderboard": "false"
  },

When this is present, ALL submissions fail, when this is absent, everything works fine.

[Didayolo]

Simply adding the fact sheet entry makes the submissions fail? And they fail without running?

I am not able to reproduce the bug.

Could you share the URL to the competition?

[johann-petrak]

Yes. Here are two projects which are absolutely identical, apart from the second one having the checkbox:

• here, submissions do work: https://www.codabench.org/competitions/2629/?secret_key=662b35ac-e475-4300-9616-cb52db833872
• here, submissions DO NOT work: https://www.codabench.org/competitions/2628/?secret_key=86205c9b-44f4-43cb-b802-118ad72432b5

Here is a sample submission file: submission.zip

[Didayolo]

Logs of the error in Django:

django_1  | Internal Server Error: /api/submissions/
django_1  | Traceback (most recent call last):
django_1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
django_1  |     response = get_response(request)
django_1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
django_1  |     response = self.process_exception_by_middleware(e, request)
django_1  |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
django_1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
django_1  |   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
django_1  |     return view_func(*args, **kwargs)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 116, in view
django_1  |     return self.dispatch(request, *args, **kwargs)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 495, in dispatch
django_1  |     response = self.handle_exception(exc)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 455, in handle_exception
django_1  |     self.raise_uncaught_exception(exc)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 492, in dispatch
django_1  |     response = handler(request, *args, **kwargs)
django_1  |   File "/app/src/apps/api/views/submissions.py", line 169, in create
django_1  |     return super(SubmissionViewSet, self).create(request, *args, **kwargs)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/mixins.py", line 20, in create
django_1  |     serializer.is_valid(raise_exception=True)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 236, in is_valid
django_1  |     self._validated_data = self.run_validation(self.initial_data)
django_1  |   File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 437, in run_validation
django_1  |     value = self.validate(value)
django_1  |   File "/app/src/apps/api/serializers/submissions.py", line 180, in validate
django_1  |     elif value not in fact_sheet[key]['selection'] and fact_sheet[key]['selection']:
django_1  | TypeError: 'in <string>' requires string as left operand, not bool

[Didayolo]

In the YAML file of the bundle, the key confirm duplicated, this may have caused the issue:

fact_sheet: {
   "confirm": {
     "key": "confirm",
     "type": "select",   
     "title": "Please confirm that you are submitting to Subtask 1, Closed track",
     "selection": ["", "    ", "I confirm"],
     "is_required": "true",
     "is_on_leaderboard": "false"
  },
  "confirm": {
     "key": "confirm",
     "type": "checkbox",  
     "title": "Please confirm that you are submitting to subtask 1, CLOSED track",
     "selection": "false",
     "is_required": "false",
     "is_on_leaderboard": "false"
  },
[...]

[johann-petrak]

No this also happens is there is no duplicate key (this was originally the case, the shared project was a modification of what I did to deal with the bug). Here is a project that does not have the duplicate but still shows the problem: https://www.codabench.org/competitions/2641/?secret_key=59048e76-3215-4a21-8421-cccc40ed28db