search

COSC1127-AI / pacman-contest-cluster

Script to run the Conquer The Flag PACMAN contest
http://ai.berkeley.edu/contest.html
Apache License 2.0
19 stars 4 forks source link

Play On-Demand Simple Contests #40

Closed ssardina closed 4 years ago

ssardina commented 6 years ago

Original report by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Build a python script as a server listening for requests:

receive a request (from TeamX) to play TeamX vs TeamY, and return a link to the log, replay file and any other info (zip):

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

I am considering the complexity of implementing the request mechanism with a web-service. This comment will sound like me thinking aloud.

I am not sure how to integrate Flask with the existing engine, so I would opt for a separate code-base entirely.

We cannot assume requests through the web are genuine, so we need to authenticate students/teams (students could maliciously exhaust the opponents' daily budget).

A separate code-base means that the configuration (i.e. teams composition) needs to be stored in the second code-base, which makes for a suboptimal admin experience. Also, students need to be given a password to authenticate, which is the second one if we consider Github; this makes for a poor user experience.

A better solution would re-use the channel that is already in place, i.e. Github.

I am thinking that, if you are willing to compromise on the "on-demand" part of this feature, we could avoid the issue entirely by simply scanning all the repos regularly (e.g. hourly? every 20 mins?), looking for a particular file in which the students can write their requests.

In order to make sure we do not re-run the same request twice (e.g. if students forget to remove their request file), we can require a timestamp to be part of the request, so we can store those and compare new requests with a list of the ones we run previously.

Or even simpler, the students could open an issue in their own repo in which to request matches, using specific formats for the issue title and body. This would avoid polluting the students code-bases and would elegantly take care of the re-running problem - Github already has identifiers for issues. Granted, it's an unorthodox use of Github issues.

What are your thoughts?

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Hi Marco, before commiting to one of the solutions you proposed (several good ones :) I have a question:

Can we use the same credentials serviced by github? If we can, this will protect us from malicious calls. The page will only show up to github users that are in the current tournament.

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Hi Nir, can you explain further what you have in mind? I am all for not giving students double credentials.

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Hi Marco,

In this linke github auth you can see in section "Authentication" how to use the credentials from github. We might just use their github credentials to login into our online server to play on demand. Make sense?

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Ah, you mean using Open ID! I get it now. I will look into it today.

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

I created a new repo to handle the requests. At the moment, it is bare-bones and only handles the authentication with Github using OAuth2. I will work on the other features on the next few days.

Next tasks:

  1. Ask for an opposing team
  2. Check against list of students
  3. Check student budget
  4. Run command to start on-demand match
  5. Gather results and deliver them (email? other server?)

Am I forgetting something?

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Thanks Marco!

The list of tasks in that order are perfect.

In terms of saving the results, a simple option would be to generate a gist, and return the handle. The gist can contain all the output. Alternatively https://transfer.sh/ can do the job, but gists will be the easiest, as I don't think github will discontinue them.

Cheers,

Nir

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Thanks Nir.

Seba, you mentioned submission is done via Git, but I could not locate such code in the repository (master branch). Can you point me in the right direction?

ssardina commented 6 years ago

Original comment by Sebastian Sardina (Bitbucket: [Sebastian Sardina](https://bitbucket.org/Sebastian Sardina), ).

Hi Marco. Here is the repo:

https://bitbucket.org/ssardina-teaching/git-submissions-tools

it is still not fully clean-up, the script is git-assignment-submissions.py and is already quite powerful and works well.

BTW, why do you need that? This will clone the student repo in your local machine. Why do you want that?

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Great! Which is the script that looks for all the repos that forked? How do you build the .csv file?

Thanks!

ssardina commented 6 years ago

Original comment by Sebastian Sardina (Bitbucket: [Sebastian Sardina](https://bitbucket.org/Sebastian Sardina), ).

The link in my previous reply is the pythons script to process the repos. The input is a csv file with the teams. The way I build that csv file is with a Google Form like this:

https://docs.google.com/forms/d/e/1FAIpQLSeNQvUgnYeNmt63UFc3Dv2trnm6_R3ed0tkIghIp-UDEUOCig/viewform

then you can just download the spreadsheet and done! works amazing.

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

We need permission to access it :) I thought that your script looked at all the repo's that forked your project and added them into a csv.

ssardina commented 6 years ago

Original comment by Sebastian Sardina (Bitbucket: [Sebastian Sardina](https://bitbucket.org/Sebastian Sardina), ).

No, the team registration is apart, there are many reasons for that, I can explain it later. Checking the forks will be very error prone.

I just realized the form is an RMIT form. :-( I just added you as a collaborator. Do not change anything but you can see how I did it, and you can make a copy of it using your gmail account

You can do the same thing with a Microsoft account (does it have forms?)

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Hi, I added the missing steps, but I have to leave now and I have not tested the last few commits, as I don't have handy a configuration file for the pacman-contest repository. If you are willing to test this, that would be great. Leave feedback here :)

You need to use the separate_html_generator branch of the pacman-contest repository (please, merge after some testing), as I have made some minor edits for compatibility with this repo. Crucially, you need to make sure the folder of the pacman-contest is named "pacman_contest", as I cannot import it here otherwise.

Other comments: the software at the moment is blocking, meaning that after submitting the run, the page will show as "loading" until the contest is run and the results are uploaded. It should not be hard to change this and send the links via email, but that may not be necessary: it depends on how long this actually takes (again, I have not tested it).

Also: at the moments, the teams composition is read from the JSON config file. This can also be changed, if you wish.

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

I just noticed that you will also need to change the pipeline, as the renamed folder and module will not be found otherwise.

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Thanks @marcotama ! I'll get to test it next week on Monday. I'll post the feedback in this thread. Cheers

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Hi @marcotama ! I started setting up the pacman-on-demand but haven't succeded yet. I've changed the "server_base_url": "43.240.96.50" but when I run

python main.py --web-server

I get

Where does it get the server_base_url? Since it's a virtual machine in nectar I cannot access the localhost address.

Any tip on how to change the url? I've created the oauth in github using the following address: http://43.240.96.50 and inserted the client id and secret.

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Hi @nirlipo , have you tried accessing using its public ip? Try running dig +short myip.opendns.com or dig +short myip.opendns.com @resolver1.opendns.com to find it.

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

Cool command :)

I get:

$ dig +short myip.opendns.com @resolver1.opendns.com 43.240.96.50

If I give you access to the server, can you try to set it up?

Cheers,

Nir

ssardina commented 6 years ago

Original comment by Sebastian Sardina (Bitbucket: [Sebastian Sardina](https://bitbucket.org/Sebastian Sardina), ).

@nirlipo , I am not following this issue as I am flat out and not teaching AI this semester, but please document everything that is needed to use this so that next year we can re-use this without having to seek around Marco or remember what was done! :-)

ssardina commented 6 years ago

Original comment by Nir Lipovetzky (Bitbucket: nirlipo, GitHub: nirlipo).

That will be the outcome. So far I just setup the server following the readme in the other repo: play-on-demand. The problem is that I'm not sure that the json configuration file is setting up correctly the ip. Or maybe there's a different problem. @marcotama Is it possible to check if the server is listening at least to the localhost:5000 from the bash command line? I've open the ports explicitly already.

Cheers,

Nir

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Hi Nir, sure thing, send me the details including user, pass, ip and json. I will try to have a look at it tomorrow after work. I'm not sure I understand the issue, and some time has passed so I don't remember all the details anymore. Sorry about this!

ssardina commented 6 years ago

Original comment by Marco Tamassia (Bitbucket: marcotama, GitHub: marcotama).

Hi Nir can you setup the contest by itself? Then I can test the on-demand feature on it. Please include 3 teams (from past contests?).

PS: the email delivery is not done yet. I will get onto it when I can easily test it.