CLI Tool

[theonejb]

GoChallenge CLI

The requirement is to create a web site and associated tools to allow easy participation in the GoChallenge. This issue is to discuss a CLI tool that can be used by the participants to interact with the site.

Right now, the process for the GoChallenge is:

• The problem is posted to the site
• The participant downloads a file or clones a repo that contains a few test cases
• The participant writes code to make those test cases pass
• The entire Go package (the code written by the participant and the original files) is zipped up and emailed to the organiser, who then manually checks the submission and marks it as successful or failed

Taking inspiration from the CLI tool used by the Exercism.io site, the GoChallenge CLI should at least:

• Allow the participant to connect their installation of the tool to their account on the GoChallenge web site
• Download the files for the currently running challenge task
• Once the participant has written code that can pass the test cases, the tool should allow them to submit it to the GoChallenge website; after making sure the test cases are passing. While not needed, ensuring that the test cases pass before submitting the code avoids accidental submissions

Once the code has been submitted, there should appear on the users account info on the GoChallenge site a submission entry with links to download the associated files. While the challenge is still on-going, only the challenge organiser or the participant should be able to download that file. Once the challenge has been completed, the file should be visible to everyone, so others can look and learn from the code.

We might also want to include functionality on the web site to automatically run the submitted code and verify that the test cases do indeed pass. In this manner, the organiser will not need to go through the submissions manually and can verify their correctness later on, towards the end of the challenge. But given the security implications of running untrusted code automatically, we might want to hold off on this. For now, I think an email to the organiser when a participant submits their code should be fine.

Please add any required feature that I might have missed, and lets start a discussion on this so we can have a tool that will be really useful to others.

[theonejb]

@sausheong @morhekil @IndianGuru Looking forward to comments and advice from you guys on this.

[sausheong]

I'm not too sure about a CLI tool. Instead of that perhaps we can have this flow:

1. The programmer reads the challenge on the site, decides to accept the challenge and click on a 'accept challenge' button
2. The programmer will be asked to log into GitHub (this is not necessary if they're already logged in)
3. GoChallenge will automatically fork the challenge into the programmer's GitHub account as a new repo
4. The programmer works on the challenge and can either push or not push their code up to their own repo
5. The programmer's repo should contain the necessary tests and integrate with a CI tool (like Travis CI)
6. GoChallenge will call automatically call the CI tool to run the tests and make sure all is passed on the final date of the challenge

I'm sure we can add other features in, but doing more integration with GitHub seems an easier way than a CLI tool. Just my 2 cents.

[theonejb]

We should discuss the other points. But one immediate issue i thought of is that if the programmer forks the repo and pushes his solution to it, its going to be visible to everyone else, since most people will have a free github account. Unless the challenge is moving away from its current competition style, that might not be what the organisors will want. Maybe theres a way to solve this issue?

On Tuesday, March 10, 2015, Sau Sheong Chang notifications@github.com wrote:

I'm not too sure about a CLI tool. Instead of that perhaps we can have this flow:

1. The programmer reads the challenge on the site, decides to accept the challenge and click on a 'accept challenge' button
2. The programmer will be asked to log into GitHub (this is not necessary if he's already logged in)
3. GoChallenge will automatically fork the challenge into the programmer's GitHub account as a new repo
4. The programmer works on the challenge and can either push or not push his code up to his own repo
5. The programmer's repo should contain the necessary tests and integrate with a CI tool (like Travis CI)
6. GoChallenge will call automatically call the CI tool to run the tests and make sure all is passed on the final date of the challenge

I'm sure we can add other features in, but doing more integration with GitHub seems an easier way than a CLI tool. Just my 2 cents.

— Reply to this email directly or view it on GitHub https://github.com/GoChallenge/gochallenge/issues/2#issuecomment-77961135 .

Asad Jibran Ahmed surfer.a1@gmail.com http://blog.asadjb.com

[sausheong]

We can fork the challenge repo but the programmer doesn't necessarily need to push their code up to GitHub.

[theonejb]

That would solve this. But how should the programmer deliver their solution?

On Tuesday, March 10, 2015, Sau Sheong Chang notifications@github.com wrote:

We can fork the challenge repo but the programmer doesn't necessarily need to push his code up to GitHub.

— Reply to this email directly or view it on GitHub https://github.com/GoChallenge/gochallenge/issues/2#issuecomment-77962704 .

Asad Jibran Ahmed surfer.a1@gmail.com http://blog.asadjb.com

[morhekil]

Coming back to the idea of exercism-like workflow, their CLI tool is being used to both pull the problem, and push the solution. It doesn't touch git toolset at all, you just do exercism fetch first to get the problem setup, and when you're ready to submit your code - do exercism submit <codefile> to push it online.

We can easily build on top of this, just by removing their choice to submit a single file only, and submitting the current directory instead (with some file mask, say *.go).

This solves the problem of solution visibility (it won't be public until it has to), and even allows people not familiar with git tools (e.g. mercurial users) to participate just as well.

PS: exercism's help pages on Fetching Exercises, Submitting Exercises to give you more info on this.

[sausheong]

@theonejb the programmer can push his code up when he wants to deliver his solution. Alternatively we can ask for sponsorship from GitHub for a 1 month private repo for each participant.

Having said that I'm ok with either solution.

[idawes]

I'm with @theonejb && @morhekil on the CLI tool approach. It should make it easy for participants to do what they need to do, and easy for organizers to deal with submissions.

[morhekil]

I was just poking around exercism's code to see how they approach various design decisions, and apparently they've rewritten their first Ruby CLI in Go already - here is the repo.

Based on me quickly skimming the code, I think it might be a good idea to leverage their ideas, and maybe even build on top of their tool - it should give us the good basis, and the only thing that really needs to be changed is a "submit" command, as theirs assumes a single-file submission only.

Thoughts?

PS: I wonder if there's any interest from exercism team to join forces to build a unified CLI tool that could be used for both exercises - not sure who to talk to there about it, so pinging @kytrinyx maybe?

[sausheong]

That's a great idea. But where does the exercism's CLI push their data to? Their servers?

[morhekil]

@sausheong yes, but their code is open, and I believe this is the repository. The API is quite simple, and we can use the same idea of providing challenge code as submodules (see here for their structure). It means that challenge authors just need to make a public repo with the challenge setup, which then can be added to the API repo as a submodule, and become available to CLI from there on.

[theonejb]

@morhekil Great idea. It would be awesome if we could use the exercism tools. They're solid and quite close to what we need. And we'll finish quicker. :+1:

[theonejb]

@morhekil I've looked at the code for their CLI tool and their API in more detail, and I have a few doubts about going with their tools for Gochallenge. Their API and CLI design seems to have a lot of features, which I'm not sure we'll need.

I thought maybe we could create something very simple that could download and upload zip files of Go packages for the current challenge. Maybe the API would have an endpoint like /v1/challenge/current/ that would return the URL to download the current challenge test cases, and that URL would be something like /v1/challenge//. This way, we have a way to get both the current challenge and any past challenges as well.

As far as submissions go, I was hoping for something just as simple. Like an API endpoint that would authenticate the user with their API key and then just store their uploaded file on some storage backend. From there, the web app would take responsibility of allowing code reviewers to accept/reject a submission, get a list of all submissions for a challenge, etc...

I know that this approach is very simple; maybe even stupidly simple, but this does translate well to the way the challenge was held currently. Plus, I'm thinking that modifying the exercism tools to work with our workflow might take as long or even a bit longer than creating something like this.

Looking forward to what you guys think. Maybe I can code up a quick demo of the CLI app and a dumb static API to see how we all feel about using it and compare it to our experience with the exercism tool?

What do you think?

[kytrinyx]

Hey, all. I think that using the exercism stuff is probably overkill, though you are welcome to use whatever we have there, should it be of any use. We're about to add multi-file support (an old issue, and fairly trivial, but there's just so much to do, and we never got to it: https://github.com/exercism/cli/issues/35)

Based on the discussion above, and my experiences with exercism, I would raise two points of caution, both related to the barrier of entry:

1. Be careful about introducing a specific version control system as a requirement for participating. Not everyone is familiar with these.
2. If you introduce a CLI, now people have to understand PATH. Once you understand it, it seems so blindingly obvious, but before you do it's completely bewildering.

I made the choice to go with a CLI for exercism, and frankly it's the hardest part of exercism for many beginners, to the point where I suspect that many people give up.

[theonejb]

@kytrinyx Thanks for advice. And also, amazing stuff with exercism!!! :+1:

I agree with point 1, about having a specific version control. But if the CLI tool isolates the user from it, does it matter? I would think that the tool would become an abstraction over the SCM?

I'm a bit confused about 2. If the participant already has Go installed (which they need for competing in the challenge anyway), isn't it necessary for them to have the GoPath already setup? And if it is, then shouldn't a go install PATH_TO_TOOL be enough for the CLI to work, without requiring the user to tamper with their PATH?

[kytrinyx]

Yeah, a go install PATH_TO_TOOL would actually solve that nicely. I forgot that with a Go challenge you don't have to cross-compile the binary, since everyone would be able to build it themselves.

[idawes]

The tool should definitely abstract from the back end storage. I'm not entirely convinced that the primary challenge storage has to be version controlled. A simple keystore might be sufficient.

On Tuesday, March 10, 2015, Asad Jibran Ahmed notifications@github.com wrote:

@kytrinyx https://github.com/kytrinyx Thanks for advice. And also, amazing stuff with exercism!!! [image: :+1:]

I agree with point 1, about having a specific version control. But if the CLI tool isolates the user from it, does it matter? I would think that the tool would become an abstraction over the SCM?

I'm a bit confused about 2. If the participant already has Go installed (which they need for competing in the challenge anyway), isn't it necessary for them to have the GoPath already setup? And if it is, then shouldn't a go install PATH_TO_TOOL be enough for the CLI to work, without requiring the user to tamper with their PATH?

— Reply to this email directly or view it on GitHub https://github.com/GoChallenge/gochallenge/issues/2#issuecomment-78115938 .

[morhekil]

@kytrinyx thanks for joining in! And I think (echoing @theonejb thoughts probably) that the concerns around using the command line and understanding PATH are much less for a Go-specific challenge, as both of these things are somewhat of a required basic knowledge from participants, as they need to sort those out to get working Go development environment itself. I do agree about binding the challenge to a specific VCS, though, and the CLI tool sounds like a nice way to abstract from that.

@idawes if we go with CLI talking to HTTP backend - this will be abstract the challenge data storage from the tool itself, and it's up to the backend then to organise it. Could you chime in with your thoughts in #4 ?

@theonejb if we're disregarding exercism's code, I'm all in with your approach. Maybe let's start with a proof of concept that would allow to fetch the first challenge?

[idawes]

I think @theonejb's suggestions sounded pretty reasonable. Let's try the stupid simple approach unless we can clearly articulate a need for something more complex.

On Tue, Mar 10, 2015 at 3:32 PM, Oleg Ivanov notifications@github.com wrote:

@kytrinyx https://github.com/kytrinyx thanks for joining in! And I think (echoing @theonejb https://github.com/theonejb thoughts probably) that the concerns around using the command line and understanding PATH are much less for a Go-specific challenge, as both of these things are somewhat of a required basic knowledge from participants, as they need to sort those out to get working Go development environment itself. I do agree about binding the challenge to a specific VCS, though, and the CLI tool sounds like a nice way to abstract from that.

@idawes https://github.com/idawes if we go with CLI talking to HTTP backend - this will be abstract the challenge data storage from the tool itself, and it's up to the backend then to organise it. Could you chime in with your thoughts in #4 https://github.com/GoChallenge/gochallenge/issues/4 ?

@theonejb https://github.com/theonejb if we're disregarding exercism's code, I'm all in with your approach. Maybe let's start with a proof of concept that would allow to fetch the first challenge?

— Reply to this email directly or view it on GitHub https://github.com/GoChallenge/gochallenge/issues/2#issuecomment-78163041 .

[idawes]

Just to get the ball rolling, Here's a first hack at what I'm thinking for the CLI:

gochallenge init --key=API_KEY gochallenge fetch current gochallenge submit

Thoughts?

[IndianGuru]

Do address this issue - https://twitter.com/paddyforan/status/575512731002007552

[idawes]

I've gone through and modified the language on the comments for this issue to be gender neutral. I think I've caught them all... anybody care to verify?

[morhekil]

@idawes maybe configure instead of init? Looks good to me otherwise, let's go ahead with it! :+1:

[paddycarver]

Looks like the original post got them all. A few comments still have some (Ctrl+F for "him"). Not sure if that's supposed to be the case or not, and I lack the skills to unpack the meaning properly. But :+1: for at least caring about supporting non-hims. :)

[idawes]

@morhekil: Sure, configure would work.

[theonejb]

Awesome. I'll cook up a basic demo tonight and we can start seeing how we feel using that.

Could someone here help with the API? Just a very basic thing that accepts auth as a header "Auth-ApiKey" and has the three endpoints which we talked about:

• /v1/challenge/current/
• /v1/challenge//
• /v1/challenge//submit/

And just save the submissions to disk? Possibly have an endpoint that can list those submissions as well with download links?

[idawes]

I'd love to, but there's a couple of caveats:

1. This would be the first API I'd ever authored... so it's going to take me a bit to figure things out
2. I'm on business travel the next couple of days, so it's going to be a while before I even get a chance to start figuring things out.

[morhekil]

@theonejb @idawes let's move API-related discussion to #4 . @theonejb , I copied your API suggestions there. I'll give a shout out on the channel to see if anyone's keen to start working on that, if not - I might be able to put a few hours down on the weekend to flesh things out

[mathcunha]

The "gochallenge submit" should build a tarball from the current directory, this file would have only the source code. After this process, the cli should return the tarball hash (e.g. md5), and then submit it. This file could be kept at the client side, just to ensure that the correct content was submitted.

At the server side, the service should store the tarball, recalculate the hash and return its value inside the confirmation response.

The evaluators could use the hash information to guarantee to the challengers that the right code was reviewed.

[theonejb]

@mathcunha I like that idea. Having data integrity checks is always good. We were thinking of zipping up the current directory. So I looked it up in the ZIP page on Wikipedia, and it turn out ZIPs already have CRC checks in them for ensuring data integrity. Maybe that would be good enough? Unless we need to calculate a hash for other reasons?

[morhekil]

I support @mathcunha idea, and I'd say it's not that much an integrity check tool, as a submission identification tool. By publishing the hash, we basically identify the exact data set that we have accepted - think of it as a commit id in git history.

[kytrinyx]

zip is easier to support on windows than gzip+tar (unless this is built into Go--I haven't checked that).

[morhekil]

@kytrinyx they're all in Go standard library, so we should be able to hide any complexity in either case: archive/tar, archive/zip, compress/gzip

[kytrinyx]

Ah, that helps. Thanks!