search

trampgeek / jobe

jobe is a server that runs small programming jobs in a variety of programming languages
MIT License
112 stars 80 forks source link

Specify langs on the command line. Meaningful exit code. #7

Closed timhunt closed 8 years ago

timhunt commented 8 years ago

Here are some changes to testsubmit.py

They let you specify which tests to run on the command-line, and they set the programs exit code, so scripts can easily tell if all the tests passed. I did this as part of my attempts to get the CodeRunner tests running on Travis-CI.

Please feel free to criticise. I am a novice Python programmer.

trampgeek commented 8 years ago

Thanks Tim. What sort of problems did you have with the Jobe install? Anything I should know about?

timhunt commented 8 years ago

I am still beating my head against it (on and off). Trying to get a CI server to work is a bit like tying to do CodeRunner with deferred feedback behaviour!

The short answer is that I don't really know what I am doing yet, and Travis, while being a very nice free things, is a bit irritating. Long answer follows:

My best shot so far is https://travis-ci.org/timhunt/moodle-qtype_coderunner/builds/174015264

One problem was that travis offers a choice between old and very old Ubuntu. That is Trusty (14.04.4) or Precise (12.04). I am using the more modern of those, but still it does not have pylint3.

https://github.com/moodlerooms/moodle-plugin-ci is very helpful for the Moodle side of things, but of course we also need JOBE. JOBE needs PHP running in Apache, and it took me a while to work out that https://docs.travis-ci.com/user/languages/php/#Apache-%2B-PHP is correct for Ubuntu 12.04 (Apache 2.2) but 14.04 has Apache 2.4 which needs slightly different config. Also Travis puts things in funny places (wwwroot under /home/travis), but fortunately that does not break the JOBE installer.

I now have JOBE mostly working: https://travis-ci.org/timhunt/moodle-qtype_coderunner/builds/174489072. Some of the remaining problems are Pylint3 missing. Other failures I have not had time to investigate properly.

I am assuming that when I next have time and inclination to have a go, I can:

I am happy to chip away at this a bit at a time. However, if you are interested, my work-in-progress is on https://github.com/timhunt/moodle-qtype_coderunner/commit/travis. Note that lots of bits are currently commented out, to minimise the time to get to testsubmit.py results. Feel free to experiment if you are interested.

trampgeek commented 8 years ago

I hadn't ever used travis-ci before. Just had a play now - I set up my own travis branch, copied your scripts in and got to the same state as you.

I think pylint ought to work fine with Trusty - our servers ran Trusty for a couple of years. I suspect the trouble is with the configuration of pylint. After installing it, you should run (as root)

pylint --reports=no --generate-rcfile > /etc/pylintrc

However, you can't simply /sudo/ that command as you get an immediate 'no access' error on the output redirect. An alternative, which should work, is

pylint --reports=no --generate-rcfile >pylintrc

sudo mv pylintrc /etc/pylintrc

I tried adding those two lines to .travis.yml, but I still get "No config file found, using default configuration" when pylint next runs. Presumably this is because travis installs things in interesting places. With some persistence this could undoubtedly be solved. Or you could just remove pylint from the whole testing process, as it's irrelevant to most users (probably). But ...

Then there's the much more serious problem that all the C compiles are aborting with the message

gcc: error trying to exec 'cc1': execvp: No such file or directory

This is a show stopper. One possibility is that it's an environment variable problem, relating to travis' different paths. The Jobe server uses the runguard sandbox, which discards all environment variables except PATH to eliminate various attack vectors.

Fixing this problem is likely to take some time. However, you could check if runguard's environment variable cleaning is the problem by tweaking /runguard/runguard.c around line 530.

I like the idea of travis-ci but there's too much head-banging involved for me. However, please keep me posted on developments.

Richard

On 10/11/16 10:19, Tim Hunt wrote:

I am still beating my head against it (on and off). Trying to get a CI server to work is a bit like tying to do CodeRunner with deferred feedback behaviour!

The short answer is that I don't really know what I am doing yet, and Travis, while being a very nice free things, is a bit irritating. Long answer follows:

My best shot so far is https://travis-ci.org/timhunt/moodle-qtype_coderunner/builds/174015264

One problem was that travis offers a choice between old and very old Ubuntu. That is Trusty (14.04.4) or Precise (12.04). I am using the more modern of those, but still it does not have pylint3.

https://github.com/moodlerooms/moodle-plugin-ci is very helpful for the Moodle side of things, but of course we also need JOBE. JOBE needs PHP running in Apache, and it took me a while to work out that https://docs.travis-ci.com/user/languages/php/#Apache-%2B-PHP is correct for Ubuntu 12.04 (Apache 2.2) but 14.04 has Apache 2.4 which needs slightly different config. Also Travis puts things in funny places (wwwroot under /home/travis), but fortunately that does not break the JOBE installer.

I now have JOBE mostly working: https://travis-ci.org/timhunt/moodle-qtype_coderunner/builds/174489072. Some of the remaining problems are Pylint3 missing. Other failures I have not had time to investigate properly.

I am assuming that when I next have time and inclination to have a go, I can:

  • Get it installing pylint from source.
  • debug the failing tests (both testsubmit.py and unit tests)

I am happy to chip away at this a bit at a time. However, if you are interested, my work-in-progress is on timhunt/moodle-qtype_coderunner@travis https://github.com/timhunt/moodle-qtype_coderunner/commit/travis. Note that lots of bits are currently commented out, to minimise the time to get to testsubmit.py results. Feel free to experiment if you are interested.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/trampgeek/jobe/pull/7#issuecomment-259529597, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDNVtNWQk-pYoVCsVtGAcG6_vFjVDGsks5q8jjngaJpZM4KsUuQ.