search

trampgeek / moodle-qtype_coderunner

CodeRunner: A moodle quiz question type that runs student-submitted program code in a sandbox to check if it satisfies a given set of tests.
GNU General Public License v2.0
207 stars 120 forks source link

Plugin for having own/custom sandbox server #5

Closed mrud closed 10 years ago

mrud commented 10 years ago

For the course I am running I wrote a plugin based on the ideone, to submit code to a local sandbox server which compiles and executes the code. The idea here is that we want to use department moodle server without and don't send the code from the students to another entity.

The benefit of having a own version instead of ideone is:

I am not sure if you would be interested in it (the server is currently rather simple and supports only two languages) but I would think having a somehow easy solution to run your own sandbox server is really nice.

I put up the current version in the paste but I would need to clean it up first.

General architecture

I didn't use the SOAP interface as I had problems creating a proxy and just used a simply flask program instead.

The server side is currently really WIP and I started to adapt your class design etc. but it is rather a hack...

If you think there could be interest/it is merge-able after cleanup, I may be able to clean things a little bit up and provide a pull request. The problem is for the time being this solution is based on the needs for our installation, so I am not sure if other people would benefit from it.

trampgeek commented 10 years ago

This is really interesting Ulrich. Having an external compute server so that CodeRunner could be deployed on a departmental or institutional server was high on my to-do list. Unfortunately, as I'm teaching a summer course at present, I haven't had much developer time, so haven't got started on it yet. I'm still working on user-defined question types; almost there, if only I can get some more programming time.

For a remote computer server I was toying with two ideas:

The first option has the advantage that the Domjudge system is very robust and supports lots of languages but the webservice interface isn't ideal: the judgehost polls the webserver for runs rather than the web server pushing the job to the compute server. Also, the API is built around a programming contest model, with the judgehost being told what program ID its judging, for which it then polls the server with required test data, which it caches. It's a very different model from CodeRunner's. The second option provides a single unified environment that provides both web server and compute server and would be configured by an admin control panel. I envisaged a simple push architecture where one server simply posted a run request to the other and waited for the response. I prefer to avoid both polling and job queues if I can help it.

I'd be interested in any comments you have on that. I'm also very keen to have a look at your implementation. I'll be interested to see what api/protocol you've used for communicating with the server.

We should discuss more later, when I've looked at your code. Unfortunately, too much on today. Later in the week maybe.

trampgeek commented 10 years ago

Hi Ulrich

I like the direction you're going with this. I much prefer the simple RESTful protocol to SOAP. Your server looks very simple and elegant and I enjoyed learning about Flask :-) I see you don't have any sort of security at present; I assume you use a firewall to ensure that only the Moodle server accesses your sandbox server?

I see three separate (and important) contributions here:

  1. A RESTful API for communicating with a remote compute server
  2. A CodeRunner sandbox module that runs jobs remotely using your API
  3. A simple server implementation of the API

Tim Hunt has suggested to me that if I want other people to use CodeRunner I need to separate the compute server from the Moodle server, as you've done. [The Ideone sandbox was just a proof-of-concept, and was never intended for production use.] I also need to package up CodeRunner into a standard Moodle plug-in form, which requires both a question-type plug in and a question-behaviour plug in.

I don't see the remote compute server(s) as being part of CodeRunner. I envisage a range of different remote server options, with a separate CodeRunner sandbox interface to each one. As I indicated earlier I was considering both an interface to a DOMjudge server ("judgehost" in their terminology) and a Moodle peer-to-peer interface. Your API/sandbox/server is a third option, with the huge advantage that it actually exists, though in somewhat fledgling form.

So ... yes please, I'd like to see your sandbox module pulled into CodeRunner, preferably with some documentation of the API. You could then distribute your server separately on github, and the documentation for configuring CodeRunner sandboxes should reference that. I'll add a settings file in the next commit so administrators can configure sandbox parameters - something I should have done ages ago. The URL for your sandbox to use, plus any other connection parameters, should go in that settings file.

Many thanks for what looks like being a really nice further contribution. [My students are loving your Ace editor addition, incidentally.]

Richard

On 21/01/14 15:04, Ulrich Dangel wrote:

For the course I am running I wrote a plugin based on the ideone, to submit code to a local sandbox server which compiles and executes the code. The idea here is that we want to use department moodle server without and don't send the code from the students to another entity.

The benefit of having a own version instead of ideone is:

  • Ability to specify runtime parameters like in the runguard sandbox
  • Support file uploads
  • Faster turn around times

I am not sure if you would be interested in it (the server is currently rather simple and supports only two languages) but I would think having a somehow easy solution to run your own sandbox server is really nice.

I put up the current version https://gist.github.com/mrud/ceaba2cfca775485f760 in the paste https://gist.github.com/mrud/ceaba2cfca775485f760 but I would need to clean it up first.

General architecture

  • |/code| to upload the code, with additional, optional parameters for sandbox parameters, file upload etc.
  • |//status| status, returning same data as ideone
  • |/link/details| details like compiler info etc.
  • |/languages| supported languages

I didn't use the SOAP interface as I had problems creating a proxy and just used a simply flask program instead.

The server side https://gist.github.com/648eefea525eea956961 is currently really WIP and I started to adapt your class design etc. but it is rather a hack...

If you think there could be interest/it is merge-able after cleanup, I may be able to clean things a little bit up and provide a pull request. The problem is for the time being this solution is based on the needs for our installation, so I am not sure if other people would benefit from it.

Reply to this email directly or view it on GitHub https://github.com/trampgeek/CodeRunner/issues/5.

mrud commented 10 years ago

Hi Richard, Sorry for the late answer.

I thought about your idea with either one connection or some messagebus. I think these make a lot of sense. I choose this route as your code had already all the required bits and bolts and if I would have to restart the remote computation server or something goes south it still would be able to recover.

If possible (if you go for a longstanding connection) we should use http because of firewalls.

Yes, currently the server side is very minimalistic and a hack (no sandbox, runtime limits are not used etc) :( I just had to come up with something in a very short period. And as I don't have access to our moodle server but the compilation server deploying updates to it are rather easy. The access restriction (IP based) is currently fine with the http server but I could implement some kind of authorization.

I am really open to changing the api/rest interface/underlying concept. I just looked at the domjudge api/ml discussion and it seemed like they were more targeting programming competitions with teams etc. OTOH I thought that my api should be simple so you could use it to write a small wrapper around domjudge/idea one/something else. As I just know the idea one api I'm not sure if this interface is sufficient.

Our first year labs start today (280 students in total) and I am very interested to see how the students will accept the tests. Other TAs who have seen your plugin are also very excited to use it. Thanks again for your plugin, it will simplify our labs here a lot. Thanks again for your great work :)

Richard Lobb notifications@github.com wrote:

Hi Ulrich

I like the direction you're going with this. I much prefer the simple RESTful protocol to SOAP. Your server looks very simple and elegant and I enjoyed learning about Flask :-) I see you don't have any sort of security at present; I assume you use a firewall to ensure that only the Moodle server accesses your sandbox server?

I see three separate (and important) contributions here:

  1. A RESTful API for communicating with a remote compute server
  2. A CodeRunner sandbox module that runs jobs remotely using your API
  3. A simple server implementation of the API

Tim Hunt has suggested to me that if I want other people to use CodeRunner I need to separate the compute server from the Moodle server, as you've done. [The Ideone sandbox was just a proof-of-concept, and was never intended for production use.] I also need to package up CodeRunner into a standard Moodle plug-in form, which requires both a question-type plug in and a question-behaviour plug in.

I don't see the remote compute server(s) as being part of CodeRunner. I

envisage a range of different remote server options, with a separate CodeRunner sandbox interface to each one. As I indicated earlier I was considering both an interface to a DOMjudge server ("judgehost" in their terminology) and a Moodle peer-to-peer interface. Your API/sandbox/server is a third option, with the huge advantage that it actually exists, though in somewhat fledgling form.

So ... yes please, I'd like to see your sandbox module pulled into CodeRunner, preferably with some documentation of the API. You could then distribute your server separately on github, and the documentation

for configuring CodeRunner sandboxes should reference that. I'll add a settings file in the next commit so administrators can configure sandbox parameters - something I should have done ages ago. The URL for your sandbox to use, plus any other connection parameters, should go in that

settings file.

Many thanks for what looks like being a really nice further contribution. [My students are loving your Ace editor addition, incidentally.]

Richard

On 21/01/14 15:04, Ulrich Dangel wrote:

For the course I am running I wrote a plugin based on the ideone, to submit code to a local sandbox server which compiles and executes the

code. The idea here is that we want to use department moodle server without and don't send the code from the students to another entity.

The benefit of having a own version instead of ideone is:

  • Ability to specify runtime parameters like in the runguard sandbox
  • Support file uploads
  • Faster turn around times

I am not sure if you would be interested in it (the server is currently rather simple and supports only two languages) but I would think having a somehow easy solution to run your own sandbox server is really nice.

I put up the current version https://gist.github.com/mrud/ceaba2cfca775485f760 in the paste https://gist.github.com/mrud/ceaba2cfca775485f760 but I would need to clean it up first.

General architecture

  • |/code| to upload the code, with additional, optional parameters for sandbox parameters, file upload etc.
  • |//status| status, returning same data as ideone
  • |/link/details| details like compiler info etc.
  • |/languages| supported languages

I didn't use the SOAP interface as I had problems creating a proxy and just used a simply flask program instead.

The server side https://gist.github.com/648eefea525eea956961 is currently really WIP and I started to adapt your class design etc. but it is rather a hack...

If you think there could be interest/it is merge-able after cleanup, I may be able to clean things a little bit up and provide a pull request. The problem is for the time being this solution is based on the needs for our installation, so I am not sure if other people would benefit from it.

Reply to this email directly or view it on GitHub https://github.com/trampgeek/CodeRunner/issues/5.

Reply to this email directly or view it on GitHub: https://github.com/trampgeek/CodeRunner/issues/5#issuecomment-33341774

trampgeek commented 10 years ago

Hi Ulrich

Thanks for the thoughts. I look forward to hearing how your courses go. [I'll email you separately.] In the meantime I'll close this issue, and I hope to get a pull request from you regarding your sandbox in due course, when you think it's good to go.

Thanks for using CodeRunner.

Richard