Create challenge in EvalAI and test all features (17th Dec)

[moriartyjm]

• [x] Create two competition phases in EvalAI: open-loop and closed-loop
• [x] Update submission instructions for these two phases
• [x] Test leaderboard function for both phases
• [x] Update link to challenge overview from https://github.com/rangl-labs/netzerotc/tree/3e96022ed3f55f9b26026f082f192a20251536a5#readme (currently blocked by #24)

[masonlr]

I'm working on the VM in the RangL subscription called "freshVM": so far have:

• [x] Updated EvalAI version
• [x] Patched all of our previous development fixes
• [x] Tested round-trip submission of the IEV challenge with a random agent

Next up on the todo list is:

• [x] Connect to new managed postgres service
• [x] Add frontend branding back in
• [x] Update rangl.org and challenges.rangl.org load balancer to point to the new VM

[masonlr]

These URLs are currently active:

challengesnew.rangl.org
submissionsnew.rangl.org

They represent the latest version of EvalAI with the NZTC challenge deployed.

Once these have been tested, we will swap the DNS settings over so that

challengesnew.rangl.org becomes challenges.rangl.org
submissionsnew.rangl.org becomes submissions.rangl.org

[masonlr]

An example testing session looks like this:

# set up the evalai cli
$ conda create -n test-submission python=3.10
$ conda activate test-submission
$ pip install evalai
$ evalai --version
evalai, version 1.3.14

$ cd random_agent_submission

# build the submission image
$ make
docker build -t submission:v0.1.0 .
[+] Building 2.7s (11/11) FINISHED
...

# set the evalai url
$ evalai host -sh http://submissionsnew.rangl.org
http://submissionsnew.rangl.org is set as the host url.

# login to evalai
$ evalai login
username: masonlr
Enter password:

Logged in successfully!

# submit the solution image
$ evalai push submission:v0.1.0 --phase nztc-dev-1
...

Your docker file is successfully submitted.

You can use `evalai submission 1` to view this submission's status.

[masonlr]

The submission should then be visible in the browser:

[jia-chenhua]

I've done what I can in https://github.com/rangl-labs/netzerotc/commit/016432cfe794049c8f54241c1a417b1cd455108a. I don't have time to investigate the detailed logic of how https://github.com/rangl-labs/netzerotc/blob/016432cfe794049c8f54241c1a417b1cd455108a/environment/server.py handles env_id supplied by https://github.com/rangl-labs/netzerotc/blob/016432cfe794049c8f54241c1a417b1cd455108a/random_agent_submission/agent.py#L11 without mentioning the first part reference_environment_direct_deployment in the env_id string. It is not efficient for me to spend quite a lot of time to investigate how to do it, as I don't have experience with HTTP services. This is too time consuming, if not time wasting.

For now, I superficially duplicated the services in https://github.com/rangl-labs/netzerotc/blob/016432cfe794049c8f54241c1a417b1cd455108a/docker-compose.yml, so that each phase has its own port number, and then I updated all other related files, without going into the details of https://github.com/rangl-labs/netzerotc/blob/016432cfe794049c8f54241c1a417b1cd455108a/environment/server.py. So this current implementation could be inefficient and it should be possible to improve it to save CPU & other resources on the VM/cloud. Therefore, @masonlr , please have a look at this and see if it can be improved. Many thanks.

[masonlr]

Thanks. I'll pick this up on Monday.

This is too time consuming, if not time wasting.

Please don't spend time on this.

[masonlr]

I've done what I can in 016432c.

If there is work underway, can you make sure there is a corresponding pull request? I've made https://github.com/rangl-labs/netzerotc/pull/95 to represent this comment above.

[masonlr]

Do we have three phases?

1. Dev Phase
2. Open-loop Phase
3. Closed-loop Phase

If so, I suggest renaming the internal env folders to make this clear, i.e.:

env_dev
env_open_loop
env_closed_loop

[masonlr]

At the moment it's not clear to a contributor how reference_environment_direct_deployment fits in.

[masonlr]

This issue has been completed from a technical perspective. I suspect the the challenge phase descriptions will need updating:

They mention posting scores on Slack only for all phases, but this is likely no longer correct.

[moriartyjm]

The challenge phases text is now in https://github.com/rangl-labs/netzerotc/blob/c785657fb0542d1f1ddf5cfd51176676f071b8c2/evaluation/content/phases.html