Worked 3/4 days ago, now unable to fork..?

[jval]

One of our teachers reported this urgent issue. She has an exam in tomorrow morning but Moodle's CodeRunner suddenly broke.

CodeRunner started to give the error "the sandbox may be down". I see there was a new version from Dockerhub 4 days ago.

[jval]

I downloaded the testsubmit.py to the host running the Docker and ran it there. It gives a lot of errors. About not being able to fork...

The same which I got when I tried to run the script directly as www-data inside the container.

[jval]

I guess it's pointless to paste the output. If you just test the latest jobeinabox you will certainly see the same. It's broken. Can you please fix it soon? :) Thanks.

[trampgeek]

The container works for me - I wouldn't have pushed it if it didn't. I have it running on both Ubuntu 22 and RHEL 8 hosts. And two other users in our department are running it too.

Your test is invalid because inside the container Jobe is running on port 80. So to test inside the container you simply type the command

python3 testsubmit.py

Let me know what you get.

As a bit of background, the only significant change to the Jobe code itself is the new testsubmit.py program. However, the JobeInABox build I pushed uses a base layer of Ubuntu:22.04 instead of Ubuntu:20.04. It's possible that your host's kernel is incompatible with Ubuntu:22 (Docker containers are not as portable as people think). What is the host system's OS version?

You could build an Ubuntu:20.04 version by changing line 5 of the Docker file from

FROM docker.io/ubuntu:22.04

to

FROM docker.io/ubuntu:20.04

and building a new container. The two commands to build a new container and run it are on dockerhub.

But first I'd like to see the testsubmit output when run in the container. If that works then run it on the host with the same command you originally used (i.e. with port 4000 rather than port 80).

Richard

PS: just to make doubly-sure, I just removed the existing container and image and ran a completely new version with the command

docker run -d -p 4000:80 --name jobe trampgeek/jobeinabox:latest

It works fine.

---

From: jval @.> Sent: Friday, December 2, 2022 5:26 AM To: trampgeek/jobeinabox @.> Cc: Subscribed @.***> Subject: [trampgeek/jobeinabox] Worked 3/4 days ago, now: Connection timed out (Issue #14)

One of our teachers reported this urgent issue. She has an exam in tomorrow morning but Moodle's CodeRunner suddenly broke.

CodeRunner started to give the error "the sandbox may be down". I see there was a new version from Dockerhub 4 days ago.

Inside the container:

@.***:/var/www/html/jobe# python3 testsubmit.py --host='public.hostname.of.this' --port=4000 Supported languages: Traceback (most recent call last): File "/var/www/html/jobe/testsubmit.py", line 1123, in sys.exit(main()) File "/var/www/html/jobe/testsubmit.py", line 1116, in main return normal_testing(langs_to_run) File "/var/www/html/jobe/testsubmit.py", line 992, in normal_testing do_get_languages() File "/var/www/html/jobe/testsubmit.py", line 953, in do_get_languages ok, lang_versions = do_http('GET', resource) File "/var/www/html/jobe/testsubmit.py", line 883, in do_http connect = http_request(method, resource, data, headers) File "/var/www/html/jobe/testsubmit.py", line 765, in http_request connect.request(method, resource, data, headers) File "/usr/lib/python3.10/http/client.py", line 1282, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 975, in send self.connect() File "/usr/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/usr/lib/python3.10/socket.py", line 845, in create_connection raise err File "/usr/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out

— Reply to this email directly, view it on GitHubhttps://github.com/trampgeek/jobeinabox/issues/14, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAIM2VWNEJTBP6LLSCVAITTWLDGRTANCNFSM6AAAAAASQ7DKBY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments.

[jval]

Host system's OS version: Debian GNU/Linux 10 (buster)

Running kernel: 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux

[jval]

Supported languages:

***** HTTP ERROR **

Response: 200 OK

A PHP Error was encountered

Severity: Warning

Message: exec(): Unable to fork [gcc --version 2>&1]

Filename: libraries/LanguageTask.php

Line Number: 518

... 39 tests, 0 passed, 0 failed, 39 exceptions Checking parallel submissions in C Traceback (most recent call last): File "/var/www/html/jobe/testsubmit.py", line 1123, in sys.exit(main()) File "/var/www/html/jobe/testsubmit.py", line 1116, in main return normal_testing(langs_to_run) File "/var/www/html/jobe/testsubmit.py", line 1010, in normal_testing check_multiple_submissions(job, NUM_PARALLEL_SUBMITS, 0) File "/var/www/html/jobe/testsubmit.py", line 737, in check_multiple_submissions t.start() File "/usr/lib/python3.10/threading.py", line 935, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

[jval]

python3 testsubmit.py | head -1

Supported languages: Traceback (most recent call last): File "/var/www/html/jobe/testsubmit.py", line 888, in do_http result = json.loads(content) File "/usr/lib/python3.10/json/init.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/www/html/jobe/testsubmit.py", line 1123, in sys.exit(main()) File "/var/www/html/jobe/testsubmit.py", line 1116, in main return normal_testing(langs_to_run) File "/var/www/html/jobe/testsubmit.py", line 998, in normal_testing result = run_test(test) File "/var/www/html/jobe/testsubmit.py", line 847, in run_test ok, result = do_http('POST', RUNS_RESOURCE, data) File "/var/www/html/jobe/testsubmit.py", line 896, in do_http output(' Response:', response.status, response.reason, content) File "/var/www/html/jobe/testsubmit.py", line 712, in output print(*args, **keywords) BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'> BrokenPipeError: [Errno 32] Broken pipe

[trampgeek]

Have you tried building a container with a base layer of Ubuntu:20? You may need to change the JDK version back to 14 as well.

Tell me more about your server. Is it a standard Debian 10 install? How many cores and how much RAM?

[trampgeek]

I just fired up a stock Debian 10 VM on Digital Ocean and ran the latest JobeInABox container on it. It works fine, all testsubmit tests pass.

Have you made any other changes to the jobeinabox host system lately? Like installing SELinux?

[jval]

Upgraded the host to bullseye, now testsubmit.py from the host:

39 tests, 39 passed, 0 failed, 0 exceptions

Checking parallel submissions in C Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK Test good C hello world OK All done

Testing a submission with an excessive cputime parameter OK

[trampgeek]

Good to know. So I assume your problem is fixed. It would be nice to know what happened, but I don't think it's a bug in JobeInABox. I'll close this thread, you can reopen if you have any further information.

[jval]

Not sure if you still need the info. The host is virtualized (KVM), minimal Debian installation as a base.

Yes, our problem is fixed. The teacher reported that everything works again. Huge thanks!