Runguard error

[dipyamanroy]

Encountering the following error when running Jobe in a Docker container hosted locally: /var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start sh': Resource temporarily unavailable Try/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

[trampgeek]

• What OS is your host running?
• How much memory and how many cores does it have?
• Are you running the pre-built jobeinabox image or have you built your own? The latter is strongly recommended (see https://github.com/trampgeek/jobeinabox)

There have been a couple of threads similar to this on the coderunner forum:

1. https://coderunner.org.nz/mod/forum/discuss.php?d=229
2. https://coderunner.org.nz/mod/forum/discuss.php?d=326

In the first of these, the problems were solved by backing off to an older version of Centos. I'm not sure what happened in the second case - I sent several emails to the person concerned but did not receive a response. In my last posting I wrote:

=== begin quote === Reading still further I discovered this alarming posting by the Red Hat product manager for containers: https://www.redhat.com/en/blog/limits-compatibility-and-supportability-containers

That was a rude shock to me - I thought one of the key points of containers was that you could run any Linux version on any other Linux host. It seems I'm seriously mistaken 🙁

This morning I asked our tech staff to spin up a RHEL8 server with docker so I could see if I could replicate your issues. He replied:

There don’t seem to be any docker packages in RHEL’s repos (including EPEL), and when I go to the docker site I get this:

We currently only provide packages for RHEL on s390x (IBM Z). Other architectures are not yet supported for RHEL, but you may be able to install the CentOS packages on RHEL. Refer to the Install Docker Engine on CentOS page for details.

(From https://docs.docker.com/engine/install/rhel)

I think you should if possible try to run JobeInABox on an Ubuntu server. === end quote ===

[dipyamanroy]

Requested information

What OS is your host running?

Ubuntu 20.04 LTS

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"

How much memory and how many cores does it have?

Sufficient on both counts I'd say. It's a Dell Poweredge R740 server. Here's the details:

MemTotal:       197596540 kB
MemFree:        106710052 kB

$ lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
CPU(s):                          96
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2

Are you running the pre-built jobeinabox image or have you built your own?

Using the pre-built one. Will consider building it, if this issue isn't resolved.

---

Thoughts

Thanks for mentioning the previous issues. I did go through them, but they didn't look similar at all.

I was able to host Jobe on an AWS instance as a workaround and have my Moodle instance point to it. But due to recent changes in my organization's network policy, I'm required to host it locally. This was tried by a previous Admin and had the similar failure, with these errors specifically. I hope this can be resolved. Seems like something's off with the runguard.c program in particular.

[trampgeek]

OK, that's the first time I've seen this message on any OS other than Centos or RHEL.

I don't think the symptoms arise from a fault in runguard itself - the message just means that the OS rejected the fork system call. The whole point of runguard is to limit resource usage so that, for example, fork bombs don't bring Jobe down. So runguard just sets up various resource limits, including a limit on the number of processes. If that limit is exceeded, you get that message. But since the process limit is fairly high (at least 20 from memory) you obviously shouldn't be getting that on the first fork call - the one that sets up a controlling shell.

Could your OS be imposing additional resource constraints on the docker server?

What is the output from testsubmit?

[dipyamanroy]

Could your OS be imposing additional resource constraints on the docker server?

I'm not sure how that could be. The only other thing running currently on the server is the Moodle instance I talked about earlier. I'm a bit stumped to be honest.

What is the output from testsubmit?

Wasn't really sure how to share the entire output without it being this long. Here it is anyways:

root@2641bf88c81b:/var/www/html/jobe# python3 testsubmit.py 
Supported languages:
    c: 11.3.0
    cpp: 11.3.0
    java: 18.0.2
    nodejs: 12.22.9
    octave: 6.4.0
    pascal: 3.2.2
    php: 8.1.2
    python3: 3.10.6

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Valid Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 with stdin
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Python3 OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 runtime error
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 file I/O
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Testing use of interpreter args with Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Testing use of runargs args with Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 program with customised timeout
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 program with support files
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Valid Python3/pylint program
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Invalid Python3/pylint program
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
UTF-8 output from Python3 (will fail unless Jobe set up for UTF-8)
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test compile error C hello world OK
Test use of compileargs with C OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test runtime error C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test timelimit on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test outputlimit on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Memory limit exceeded in C (seg faults)
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Infinite recursion (stack error) on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
C program controlled forking
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
A C program with ASCII non-UTF-8-compatible output
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
Valid Octave
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
octave with stdin
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Octave (treated as runtime error) OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
Syntactically valid Nodejs hello world
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Nodejs OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Correct Php program 
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically incorrect Php program  OK
Syntactically incorrect Php program  OK
Correct Java program  OK
Correct Java program without supplied sourcefilename  OK
Syntactically incorrect Java program  OK
Java program with a support class (.java) OK
Java program with Unicode output (will fail unless Jobe set up for UTF-8)  OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C++ hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test compile error C++ hello world OK

***************** HTTP ERROR ******************

 Response: 500 Internal Server Error 
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>A PHP Error was encountered</h4>

<p>Severity: Warning</p>
<p>Message:  file_get_contents(prog.pas.err): Failed to open stream: No such file or directory</p>
<p>Filename: libraries/pascal_task.php</p>
<p>Line Number: 36</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/pascal_task.php<br />
            Line: 36<br />
            Function: file_get_contents         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 220<br />
            Function: compile           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>An uncaught Exception was encountered</h4>

<p>Type: Error</p>
<p>Message: Typed property Task::$executableFileName must not be accessed before initialization</p>
<p>Filename: /var/www/html/jobe/application/libraries/pascal_task.php</p>
<p>Line Number: 51</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 397<br />
            Function: getExecutablePath         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 160<br />
            Function: getRunCommand         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 224<br />
            Function: execute           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>

***************** HTTP ERROR ******************

 Response: 500 Internal Server Error 
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>A PHP Error was encountered</h4>

<p>Severity: Warning</p>
<p>Message:  file_get_contents(prog.pas.err): Failed to open stream: No such file or directory</p>
<p>Filename: libraries/pascal_task.php</p>
<p>Line Number: 36</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/pascal_task.php<br />
            Line: 36<br />
            Function: file_get_contents         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 220<br />
            Function: compile           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>An uncaught Exception was encountered</h4>

<p>Type: Error</p>
<p>Message: Typed property Task::$executableFileName must not be accessed before initialization</p>
<p>Filename: /var/www/html/jobe/application/libraries/pascal_task.php</p>
<p>Line Number: 51</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 397<br />
            Function: getExecutablePath         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 160<br />
            Function: getRunCommand         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 224<br />
            Function: execute           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>

39 tests, 13 passed, 24 failed, 2 exceptions

Checking parallel submissions in C

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test good C hello world OK
Test good C hello world OK
All done

Testing a submission with an excessive cputime parameter
OK

Hope this can be of some help. Thanks.

[trampgeek]

Many thanks for the helpful and prompt response.

Yes, I'm a bit stumped too. There are upwards of 2000 Jobe servers running around the world and I've only had a small handful of reports of this problem, all involving either CentOS or RHEL, which are known to be fragile when running Docker containers. A problem on an Ubuntu host is a first.

But the output from the testsubmit run is very interesting. Two odd things happened there:

1. All Java tests ran OK. Since Java is by far the most resource-hungry Jobe language, that's surprising. However, because Java is so demanding, the Java task manager on Jobe uses a very high limit on the allowed number of processes (256) and turns off the runguard memory limit checking, relying on the JVM to manage memory securely. That is probably highly significant.
2. When the tester submitted 10 C jobs in fairly quick succession, only three of them gave the "cannot start sh" message. The other 7 ran fine. When run in isolation, the identical job failed. I have no idea how this could happen.

However, the Java result is potentially very revealing. Could you check please that if you submit a Java program (e.g. classic Hello world) from CodeRunner, that Java runs OK?

Also, could you then go back to your original trivial C program and do the following, please:

1. Click customise
2. Open the Advanced Customisation section and set the MemLimit field to 0.
3. Try running the code.
4. If it worked, we're getting somewhere, though I don't know where exactly.
5. If it didn't work, set the Parameters field of the advanced customisation to the string {"numprocs": 500} (including the braces) while leaving the MemLimit set to 0.
6. Run the question again.

I look forward to hearing what happens. I'm really curious at this point and am grateful for your help with the debugging.

[dipyamanroy]

Thanks a lot for your insights!

---

However, the Java result is potentially very revealing. Could you check please that if you submit a Java program (e.g. classic Hello world) from CodeRunner, that Java runs OK?

Runs just fine.

Also, could you then go back to your original trivial C program and do the following, please:

The program failed like previously with just the MemLimit field set to 0. However with both MemLimit set to 0, and Parameters set to {"numprocs": 500}, it works perfectly!

---

Apologies for my lack of knowhow, could you briefly explain what changing the advanced customisation fields did? Thanks again for your help!

[trampgeek]

Many thanks for that. It's certainly progress. But it's not meant as a solution. You should be able to run the test suite.

To clarify: the runguard module (a very old version I took from the domjudge programming contest system) uses Linux ulimits to limit the resources a job can consume. Limits include memory, number of processes, number of open files, etc. When jobe receives a task, if first waits for a free Jobe user (there are users jobe00, jobe01, .. jobe). It then compiles and runs the job, using runguard. Runguard cleans the environment, starts a watchdog timer and sets the various ulimit values for the user. The "numprocs" parameter in the CodeRunner question authoring form leads directly to setting the RLIMIT_NPROC ulimit value, which is the maximum number of processes the appropriate jobe user can have. This limits forking.

From the tests you've just done, it would appear that in your environment the process limit has to be set extraordinarily high to get jobs to run. I need to figure out why.

A unique property of your system is the large number of CPUs - 96. Typical Jobe systems have just 8. That shouldn't be relevant because the ulimits should be applying to each jobe user as it runs each job. If the number of CPUs is relevant there must indeed be a bug in runguard, or how I'm using it. I'm using a very old version of runguard to avoid the complexities of setting up cgroups, which later versions of runguard require.

I need to pore over the runguard code to understand it better. But to further help with diagnosis, would you please set the MemLimit field back to empty and binary search to find the minimum value of numprocs that will allow the simple C job to run? If the number comes out to around 100, that will strongly suggest that the number of CPUs on your system is the key factor.

[trampgeek]

One more thing if you wouldn't mind ...

Could you please run the following command inside the container and tell me how many jobe users are set up:

cat /etc/passwd | grep jobe

Thanks

Richard

[dipyamanroy]

Thanks a lot for the explainer into what might be happening. Following your instructions on setting the MemLimit field back to empty and then trying to find the minimum value of numprocs, I have the following results:

1. The error reappears when I set numprocs to 48 (so somewhere in that ballpark).
2. A new error appears when numprocs was set to 56:
3. With numprocs set to 60, there's no error and the C job is executed as usual.

Here's what you asked next regarding the number of jobe users set up in the container:

$ docker exec -it jobe bash
root@2641bf88c81b:/# cat /etc/passwd | grep jobe
jobe:x:999:999:Jobe user. Provides home for runs and files.:/home/jobe:/bin/false
jobe00:x:998:999:Jobe server task runner:/home/jobe00:/bin/false
jobe01:x:997:999:Jobe server task runner:/home/jobe01:/bin/false
jobe02:x:996:999:Jobe server task runner:/home/jobe02:/bin/false
jobe03:x:995:999:Jobe server task runner:/home/jobe03:/bin/false
jobe04:x:994:999:Jobe server task runner:/home/jobe04:/bin/false
jobe05:x:993:999:Jobe server task runner:/home/jobe05:/bin/false
jobe06:x:992:999:Jobe server task runner:/home/jobe06:/bin/false
jobe07:x:991:999:Jobe server task runner:/home/jobe07:/bin/false
root@2641bf88c81b:/# 

---

You were suspecting the initial tests to yield a value around 100, for the number of CPUs, but that isn't so. But, continuing on that line of thought, 56 is the maximum number of cores 1 this machine has, so could that be the factor instead?

Many thanks again for responding with the explainer to the issue at hand. And thanks a lot for taking the time out of your weekend trying to debug this. :)

[trampgeek]

Thank you for the great responses to all my questions.

At this point I'm a bit baffled. You might be right that the number of cores rather than the number of CPUs is significant, but that seems even more improbable. I frankly don't see how either is relevant.

I just checked on an 8-core machine, and I can set numprocs to 1 and have most jobs in the test suite run. [A few need extra threads.] The errors you're getting are all because forks are being disallowed as a consequence of the NPROCS ulimit setting. That implies that the effective user of the task running the code already has around 50 - 60 processes/threads running.

Hmmm. How long has the jobeinabox container been up? Is it possible that somehow there are a whole lot of tasks running within it (which of course shouldn't be possible, but let's suppose). If jobe00 through jobe02 all have 50 or so stuck processes, but the other jobe users don't, that would explain how 7 of the 10 jobs ran OK when testsubmit threw 10 at it in quick succession.

Can you restart the container and rerun testsubmit, please? Or even better, delete the container and re-run the image? [But I'm about to go and get some exercise so won't be back on line for a while. And surely you need to sleep some time, too?! What time zone are you in?]

[dipyamanroy]

Hmmm. How long has the jobeinabox container been up? Is it possible that somehow there are a whole lot of tasks running within it (which of course shouldn't be possible, but let's suppose). If jobe00 through jobe02 all have 50 or so stuck processes, but the other jobe users don't, that would explain how 7 of the 10 jobs ran OK when testsubmit threw 10 at it in quick succession.

This jobeinabox container was set up new around the time I created the issue.

So, my predecessor, to get around the fact that the errors were appearing on the local machine, hosted it on an AWS instance, which got things working, but when I took over, just wanted to get to the bottom of what was happening, hence me being here :smile:.

It's interesting if the stuck processes are on the 3 users, will try to spawn some tmux sessions and monitor (if possible), with none of the changes applied.

Can you restart the container and rerun testsubmit, please? Or even better, delete the container and re-run the image?

I'll do that and get back to you.

[But I'm about to go and get some exercise so won't be back on line for a while. And surely you need to sleep some time, too?! What time zone are you in?]

Oh of course, there's no hurry. And I'm in UTC+5:30, so a little behind you, which means I already got my sleep :sweat_smile:.

Hope you have a nice weekend!

Thanks again, Roy D

[trampgeek]

Monitoring the existing container is a great idea! Let me know if you find anything.

A problem with the stuck process explanation is: "Why doesn't this happen to other people?" I realise that many people don't report issues and just give up, but there are nearly 3000 CodeRunner installs out there and nearly all should have at least one Jobe server. We have about 8 at University of Canterbury, all running Ubuntu 20:04, and have probably fired up at least 30 or 40 over the years, handling many millions of submissions. Jobe servers normally need no maintenance at all - even the occasional stuck process would result over time in error messages like you're seeing. So what's different about your environment, other than the large number of CPUs?

So I'm still baffled, but hopeful of eventually finding an explanation. I'm grateful to be communicating with some who's helpful and knowledgeable - thanks.

[dipyamanroy]

So what's different about your environment, other than the large number of CPUs?

I don't think the large number of CPUs isn't all that unusual considering this is a rack server. There's not much running on it too, just an instance of Moodle (we call it Euclid), and now Jobe, all of them containerised. I'll try to find out if there's anything that stands out, but I'll have to wait for my university to reopen Monday for that :cold_sweat:. I'll also ask for elevated access on the server to set up the monitoring properly.

So I'm still baffled, but hopeful of eventually finding an explanation. I'm grateful to be communicating with some who's helpful and knowledgeable - thanks.

Thanks for all the kind words. I'm really glad that I can be of help. As you probably would've figured by now, I'm just a student, albeit curious. I'll try to have my professor, who got our university to use Moodle and then set up CodeRunner (with one of my seniors) to join in this thread. I've recently spoken to the senior about this and I'm sure he'd want to join in as well.

Thanks again for all the help! Really means a lot.

Edit: I'll try to get back to you by early next week, with details on our environment, and if I see something on the resource monitors.

[trampgeek]

Thanks Dipyaman. No I hadn't guessed you were a student, but that probably explains the unusual helpfulness :-)

Something else you might like to know: if you submit a run from CodeRunner with the sandbox parameter debug set to 1, the directory containing the run is not deleted. It will be in /home/jobe/runs. It contains the bash command that is used to run the job via runguard, the source file, and the stdout and stderr results. However, you'd need to use a sufficiently high numprocs for this to happen, so unfortunately it would only record the successful runs, not the failed ones. But it can be useful to observe how runguard is behaving, and it makes it very easy to vary the numprocs parameter to runguard (except it's called 'nproc' there).

So the sandbox parameter field might be something like:

{"numprocs": 80, "debug": 1}

Good luck! I'd really like to know what is happening here.

-- Richard

[dipyamanroy]

Really sorry about the late reply. Continuing from your last comment, I ran the dummy C question with "debug": 1 as a parameter. As expected, here is what /home/jobe/runs/ looks like considering there were 3 test cases:

root@2641bf88c81b:/# ls -R /home/jobe/runs/
/home/jobe/runs/:
jobe_0QyRtx  jobe_IZfQvc  jobe_iu9qBT

/home/jobe/runs/jobe_0QyRtx:
__tester__.c  __tester__.c.exe  prog.cmd  prog.err  prog.in  prog.out

/home/jobe/runs/jobe_IZfQvc:
__tester__.c  __tester__.c.exe  prog.cmd  prog.err  prog.in  prog.out

/home/jobe/runs/jobe_iu9qBT:
__tester__.c  __tester__.c.exe  prog.cmd  prog.err  prog.in  prog.out
root@2641bf88c81b:/#

Going through them, here's the prog.cmd for the first test case:

root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# cat prog.cmd 
sudo /var/www/html/jobe/application/libraries/../../runguard/runguard --user=jobe00 --group=jobe --cputime=5 --time=10 --filesize=20000 --nproc=81 --no-core --streamsize=2000 --memsize=500000 sh -c './__tester__.c.exe' >prog.out 2>prog.err <prog.in
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# 

Now, keeping all the parameters intact and setting --nproc=48 led to the error coming back again, as seen in prog.err:

root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# sudo /var/www/html/jobe/application/libraries/../../runguard/runguard --user=jobe00 --group=jobe --cputime=5 --time=10 --filesize=20000 --nproc=48 --no-core --streamsize=2000 --memsize=500000 sh -c './__tester__.c.exe' >prog.out 2>prog.err <prog.in
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# cat prog.err 
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# 

You did say earlier, looking at the output of testsubmit.py that a few users (jobe00 through jobe02 may have stuck processes). So, switching the user to jobe04 (--user=jobe04) the program ran with --nproc=48 as you can see below:

root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# sudo /var/www/html/jobe/application/libraries/../../runguard/runguard --user=jobe04 --group=jobe --cputime=5 --time=10 --filesize=20000 --nproc=48 --no-core --streamsize=2000 --memsize=500000 sh -c './__tester__.c.exe' >prog.out 2>prog.err <prog.in
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# cat prog.err 
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# cat prog.out
5
root@2641bf88c81b:/home/jobe/runs/jobe_0QyRtx# 

We seem to have gotten very close to the issue.

Is there a way to find out these stuck processes, or just access the users created by Jobe? Because if we are able to, then one should be able to remove any bottlenecks put on these specific users.

Once again, thanks so much for all the pointers. Hope this helps!

[trampgeek]

Many thanks Dipyaman,

Indeed it's interesting to have proof that jobe04 runs a task that jobe00 can't run. But if there really are stuck processes, what are they and why are they there? Jobe should clean up all processes at the end of a run; I've never seen stuck processes before.

If you run the command

ps alx | grep jobe

do you see any rogue processes hanging around?

Also, I'd like to see what happens if you fire up another jobe container (with a different name and on a different port), and run testsubmit.py in that container. Do you have the required rights to do that? The command would be something like

$sudo docker run -d -p 5000:80 --name jobecopy trampgeek/jobeinabox:latest

[dipyamanroy]

If you run the command

ps alx | grep jobe

do you see any rogue processes hanging around?

$ ps alx | grep jobe
0  1004 2388991 1328707  20   0   8908  2668 pipe_r S+   pts/4      0:00 grep jobe
$ 

Nothing appears to be unusual... for this kind of a machine.

Also, I'd like to see what happens if you fire up another jobe container (with a different name and on a different port), and run testsubmit.py in that container. Do you have the required rights to do that?

Seems to be yielding similar results again. I didn't really check for differences :sweat_smile: . Here's the output:

root@d9b30c6bf549:/var/www/html/jobe# python3 testsubmit.py 
Supported languages:
    c: 11.3.0
    cpp: 11.3.0
    java: 18.0.2
    nodejs: 12.22.9
    octave: 6.4.0
    pascal: 3.2.2
    php: 8.1.2
    python3: 3.10.6

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Valid Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 with stdin
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Python3 OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 runtime error
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 file I/O
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Testing use of interpreter args with Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Testing use of runargs args with Python3
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 program with customised timeout
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Python3 program with support files
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Valid Python3/pylint program
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Invalid Python3/pylint program
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
UTF-8 output from Python3 (will fail unless Jobe set up for UTF-8)
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test compile error C hello world OK
Test use of compileargs with C OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test runtime error C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test timelimit on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test outputlimit on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Memory limit exceeded in C (seg faults)
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Infinite recursion (stack error) on C
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
C program controlled forking
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
A C program with ASCII non-UTF-8-compatible output
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
Valid Octave
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
octave with stdin
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Octave (treated as runtime error) OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 12, 'cmpinfo': '', 'stdout': '', 'stderr': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n"}
Syntactically valid Nodejs hello world
Jobe result: Runtime error

No output

Error output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically invalid Nodejs OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Correct Php program 
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Syntactically incorrect Php program  OK
Syntactically incorrect Php program  OK
Correct Java program  OK
Correct Java program without supplied sourcefilename  OK
Syntactically incorrect Java program  OK
Java program with a support class (.java) OK
Java program with Unicode output (will fail unless Jobe set up for UTF-8)  OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C++ hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test compile error C++ hello world OK

***************** HTTP ERROR ******************

 Response: 500 Internal Server Error 
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>A PHP Error was encountered</h4>

<p>Severity: Warning</p>
<p>Message:  file_get_contents(prog.pas.err): Failed to open stream: No such file or directory</p>
<p>Filename: libraries/pascal_task.php</p>
<p>Line Number: 36</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/pascal_task.php<br />
            Line: 36<br />
            Function: file_get_contents         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 220<br />
            Function: compile           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>An uncaught Exception was encountered</h4>

<p>Type: Error</p>
<p>Message: Typed property Task::$executableFileName must not be accessed before initialization</p>
<p>Filename: /var/www/html/jobe/application/libraries/pascal_task.php</p>
<p>Line Number: 51</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 397<br />
            Function: getExecutablePath         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 160<br />
            Function: getRunCommand         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 224<br />
            Function: execute           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>

***************** HTTP ERROR ******************

 Response: 500 Internal Server Error 
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>A PHP Error was encountered</h4>

<p>Severity: Warning</p>
<p>Message:  file_get_contents(prog.pas.err): Failed to open stream: No such file or directory</p>
<p>Filename: libraries/pascal_task.php</p>
<p>Line Number: 36</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/pascal_task.php<br />
            Line: 36<br />
            Function: file_get_contents         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 220<br />
            Function: compile           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>
<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>An uncaught Exception was encountered</h4>

<p>Type: Error</p>
<p>Message: Typed property Task::$executableFileName must not be accessed before initialization</p>
<p>Filename: /var/www/html/jobe/application/libraries/pascal_task.php</p>
<p>Line Number: 51</p>

    <p>Backtrace:</p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 397<br />
            Function: getExecutablePath         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/LanguageTask.php<br />
            Line: 160<br />
            Function: getRunCommand         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/controllers/Restapi.php<br />
            Line: 224<br />
            Function: execute           </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/application/libraries/REST_Controller.php<br />
            Line: 793<br />
            Function: runs_post         </p>

            <p style="margin-left:10px">
            File: /var/www/html/jobe/index.php<br />
            Line: 315<br />
            Function: require_once          </p>

</div>

39 tests, 13 passed, 24 failed, 2 exceptions

Checking parallel submissions in C

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK

***************** FAILED TEST ******************

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable\nTry `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.\n", 'stdout': '', 'stderr': ''}
Test good C hello world
Jobe result: Compile error

Compiler output:
/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

************************************************

Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
All done

Testing a submission with an excessive cputime parameter
OK
root@d9b30c6bf549:/var/www/html/jobe# 

[trampgeek]

Many thanks for that. I am starting to have an inkling about what's going on, but I need to think more about it and, if I'm right, come up with a workaround. I've got too much else on at present to put the time into this, so please standby - possibly for several days. But, again, I'm very grateful for the help - you may just have helped me find a nasty bug.

[dipyamanroy]

I'm really happy to have helped! I'll stay tuned for an update.

Thanks for all your help. And thank you so much for making and continuing to maintain Jobe and CodeRunner. :smile:

[trampgeek]

Sorry about the delay. Another user has just reported a similar issue on the JobeInABox github repo, so I really need to get this resolved, to the extent possible.

I think the problem is that user namespace inside the container is shared with that on the host. TBH I wasn't even aware of that until you began this thread. I'm still trying to figure out the implications.

To help confirm whether that's the problem or not, can you tell me the output of the following two commands on the host, please:

grep ":99" /etc/passwd

ps lx -u 996,997,998

[trampgeek]

Oops, sorry. Drop the x from that last command. Should be:

ps l -u 996,997,998

[trampgeek]

I've pushed a change to Jobe to allow customised setting of the UIDs allocated to the jobe processes. I also pushed a new version of Dockerfile and updated the latest jobeinabox image to make use of the new functionality.

Are you able to check with jobeinabox:latest to confirm that the problem has been solved, please?

Thanks again for the help in reporting and debugging.

[bhanukiran]

Many thanks for fixing the issue. The latest image works without any need of setting a value for numprocs. Also the testsubmit.py check terminates without any errors. :+1:

[trampgeek]

Good to hear. Thank @bhanukiran