codalab / codabench

Codabench is a flexible, easy-to-use and reproducible benchmarking platform. Check our paper at Patterns Cell Press https://hubs.li/Q01fwRWB0
Apache License 2.0
77 stars 28 forks source link

Phase start/end do not work properly with datetime #1399

Closed johann-petrak closed 7 months ago

johann-petrak commented 7 months ago

In order to debug what will be happening when a phase change occurs I configured two phases like this:

phases:
  - name: Setup Phase
    description: '...'
    start: 2024-03-01T13:00:00+02:00
    end: 2024-04-10T13:00:00+02:00
    max_submissions_per_day: 1000
    max_submissions: 1000
    execution_time_limit: 60
    solutions: []
    tasks:
      - 0
  - name: Trial Phase
    description: '....'
    start: 2024-04-10T14:00:00+02:00
    end: 2024-04-30
    max_submissions_per_day: 20
    max_submissions: 20
    execution_time_limit: 60
    solutions: []
    # public_data: public_data
    # starting_kit: starting_kit
    tasks:
      - 0

When uploading the bundle, I get the error:

Creation failed

Phases dates conflict. Phase: Trial Phase should start after Phase: Setup Phase has ended

Obviously, the trial phase DOES start after the setup phase has ended, there is even a gap of one hour!

This seems to be rather broken: if I instead use the starting datetime 2024-04-11T14:00:00+02:00 forthe trial phase, it does work, but the indicated information in the web page is all wrong:

Note that the documentation says "start: Datetime string for the start of the competition. Isoformat strings are recommended." so a datetime (not just a date) should be allowed and indeed that would be essential to coordinate the work and for testing.

Didayolo commented 7 months ago

Related:

johann-petrak commented 7 months ago

This seems to be two bugs really: showing "Never" in the overview is a separate bug and when that happens, it is not possible to submit any submissions any more see #1400

Didayolo commented 7 months ago

This issue is a duplicate of #1155. The remaining point in #1155 is to add the possibility to use actual datetime for start and end dates of phases. Right now this feature is not implemented, only the dates are taken into account.