Connection error between judge and bridged container

Dellos7 commented 1 year ago

Hi,

I'm having issues when I try to set up my judge and connect it to the DMOJ site. I'm following instructions at https://docs.dmoj.ca/#/judge/setting_up_a_judge but my judge is not able to connect to the DMOJ site that I have set up with your docker image using docker-compose.

When I try to execute the judge as described here it is not able to connect to the bridged address. I have done all these:

Created a judge in the admin panel in the DMOJ site (name + key)
Created the judge.yml file with the judge id, key, path to the problems folder and runtime.
Executed dmoj -c judge.yml "$IP" (my PORT is 9999 as I've not changed it) and I've tried with the following $IP values:
- 127.0.0.1
- 10.0.2.15 (this is my private host IP)
- bridged (this is the BRIDGED_JUDGE_ADDRESS at settings.py in the repo folder.

I also have set up 10.0.2.15 as my HOST value at site.env and as server_name at nginx.conf.

I guess the problem is that my judge cannot connect to the bridged container but I actually don't know how to make it work.

Any help is really appreciated.

Thank you very much in advance.

Ninjaclasher commented 1 year ago

What's the error when you run dmoj -c judge.yml "$IP"?

Dellos7 commented 1 year ago

This is the error log that I get when I run dmoj -c judge.yml 10.0.2.15:

dmoj@dmoj-VirtualBox:~/DMOJ/dmoj-docker/dmoj$ dmoj -c judge.yml 10.0.2.15
Self-testing executors
Self-testing AWK:    Success [0.006s, 1392 KB]   awk 1.3.4
Self-testing C:      Success [0.002s, 1028 KB]   gcc 11
Self-testing C11:    Success [0.002s, 1028 KB]   gcc11 11
Self-testing CPP03:  Success [0.002s, 1976 KB]   g++ 11
Self-testing CPP11:  Success [0.003s, 1976 KB]   g++11 11
Self-testing CPP14:  Success [0.002s, 1976 KB]   g++14 11
Self-testing CPP17:  Success [0.003s, 1976 KB]   g++17 11
Self-testing CPP20:  Success [0.003s, 1976 KB]   g++20 11
Self-testing GAS64:  Success [0.010s, 4 KB]      as_x64 2.38, ld_x64 2.38
Self-testing PERL:   Success [0.012s, 5952 KB]   perl 5.34.0
Self-testing PY3:    Success [0.017s, 8736 KB]   python3 3.10.6
Self-testing SED:    Success [0.003s, 1132 KB]   sed 4.8
Self-testing TCL:    Success [0.004s, 4200 KB]   tclsh 8.6
Self-testing TEXT:   Success [0.012s, 1992 KB]   cat 8.32

Running live judge...
Warning: `problem_storage_root` is deprecated, use `problem_storage_globs` instead
INFO 2023-03-22 09:47:06,127 14882 packet Preparing to connect to [10.0.2.15]:9999 as: PYTHON_JUDGE
INFO 2023-03-22 09:47:06,127 14882 packet TLS not enabled.
INFO 2023-03-22 09:47:06,143 14882 packet Opening connection to: [10.0.2.15]:9999
ERROR 2023-03-22 09:47:06,144 14882 packet Connection failed due to socket error: [10.0.2.15]:9999
Traceback (most recent call last):
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 128, in _do_reconnect
    self._connect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 97, in _connect
    self.conn = socket.create_connection((self.host, self.port), timeout=5)
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
WARNING 2023-03-22 09:47:06,145 14882 packet Attempting reconnection in 4s: [10.0.2.15]:9999
INFO 2023-03-22 09:47:10,166 14882 packet Opening connection to: [10.0.2.15]:9999
ERROR 2023-03-22 09:47:10,167 14882 packet Connection failed due to socket error: [10.0.2.15]:9999
Traceback (most recent call last):
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 128, in _do_reconnect
    self._connect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 97, in _connect
    self.conn = socket.create_connection((self.host, self.port), timeout=5)
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 128, in _do_reconnect
    self._connect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 97, in _connect
    self.conn = socket.create_connection((self.host, self.port), timeout=5)
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
WARNING 2023-03-22 09:47:10,170 14882 packet Attempting reconnection in 6s: [10.0.2.15]:9999
^CTraceback (most recent call last):
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 128, in _do_reconnect
    self._connect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 97, in _connect
    self.conn = socket.create_connection((self.host, self.port), timeout=5)
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 128, in _do_reconnect
    self._connect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 97, in _connect
    self.conn = socket.create_connection((self.host, self.port), timeout=5)
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dmoj/.local/bin/dmoj", line 8, in <module>
    sys.exit(main())
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/judge.py", line 616, in main
    judge = ClassicJudge(
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/judge.py", line 534, in __init__
    super().__init__(packet.PacketManager(host, port, self, env['id'], env['key'], **kwargs))
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 87, in __init__
    self._do_reconnect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 134, in _do_reconnect
    self._reconnect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 124, in _reconnect
    self._do_reconnect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 134, in _do_reconnect
    self._reconnect()
  File "/home/dmoj/.local/lib/python3.10/site-packages/dmoj/packet.py", line 122, in _reconnect
    time.sleep(self.fallback)
KeyboardInterrupt

and this is my judge.yml file:

id: PYTHON_JUDGE
key: <my_key>
problem_storage_root:
  - /home/dmoj/DMOJ/dmoj-docker/dmoj/problems
runtime:
  as_x64: /usr/bin/x86_64-linux-gnu-as
  as_x86: /usr/bin/as
  awk: /usr/bin/mawk
  cat: /usr/bin/cat
  g++: /usr/bin/g++
  g++11: /usr/bin/g++
  g++14: /usr/bin/g++
  g++17: /usr/bin/g++
  g++20: /usr/bin/g++-11
  gcc: /usr/bin/gcc
  gcc11: /usr/bin/gcc
  ld_x64: /usr/bin/x86_64-linux-gnu-ld
  ld_x86: /usr/bin/ld
  perl: /usr/bin/perl
  python3: /usr/bin/python3
  sed: /usr/bin/sed
  tclsh: /usr/bin/tclsh

and this is the specific configuration at repo/dmoj/settings.py:

# Bridged configuration
BRIDGED_JUDGE_ADDRESS = [('localhost', 9999)]
BRIDGED_JUDGE_PROXIES = None
BRIDGED_DJANGO_ADDRESS = [('localhost', 9998)]
BRIDGED_DJANGO_CONNECT = None

and this is the configuration at repo/dmoj/local_settings.py:

BRIDGED_JUDGE_ADDRESS = [('bridged', 9999)]

# The bridged daemon bind address and port to communicate with the site.
BRIDGED_DJANGO_ADDRESS = [('bridged', 9998)]

The bridged container configuration at the docker-compose.yml:

bridged:
    build:
      context: .
      dockerfile: ./bridged/Dockerfile
    image: ninjaclasher/dmoj-bridged
    init: true
    restart: unless-stopped
    volumes:
      - ./problems/:/problems/
      - ./repo/:/site/
    working_dir: /site/
    env_file: [environment/mysql.env, environment/site.env]
    networks: [site, nginx, db]
    ports:
      - 9999:9999
    depends_on: [base, db, redis]

I have checked that the firewall ufw is disabled (I'm at Ubuntu 22.04.2 LTS). However, iptables rules are the following:

dmoj@dmoj-VirtualBox:~/DMOJ/dmoj-docker/dmoj$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (4 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:9998
ACCEPT     tcp  --  anywhere             172.19.0.3           tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (4 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Maybe problem is that port 9999 is not allowed at chain DOCKER? I'm asking having no idea, actually.

Thank you very much, hope all this information helps.

Dellos7 commented 1 year ago

Hi again @Ninjaclasher ,

I've found that dmoj-bridged and dmoj-celery containers are not able to start. Actually, they try to restart theirselves again and again. This is my log when I run docker container ls:

CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS                         PORTS                               NAMES
d54fe402ed6a   nginx:alpine                "/docker-entrypoint.…"   6 minutes ago   Up 6 minutes                   0.0.0.0:80->80/tcp, :::80->80/tcp   dmoj_nginx
ae43d28b0e51   ninjaclasher/dmoj-site      "/bin/sh -c 'uwsgi -…"   6 minutes ago   Up 6 minutes                   8000/tcp                            dmoj-site-1
c0885ad67f79   ninjaclasher/dmoj-bridged   "/bin/sh -c 'python3…"   6 minutes ago   Restarting (1) 7 seconds ago                                       dmoj-bridged-1
415d28909c93   ninjaclasher/dmoj-celery    "/bin/sh -c 'celery …"   6 minutes ago   Restarting (2) 6 seconds ago                                       dmoj-celery-1
4d6ad089ab69   ninjaclasher/dmoj-pdfoid    "/bin/sh -c 'pdfoid …"   6 minutes ago   Up 6 minutes                   8888/tcp                            dmoj-pdfoid-1
1cad02480697   ninjaclasher/dmoj-texoid    "/bin/sh -c 'texoid …"   6 minutes ago   Up 6 minutes                   8888/tcp                            dmoj-texoid-1
bc515d24fb50   ninjaclasher/dmoj-mathoid   "docker-entrypoint.s…"   6 days ago      Up 6 minutes                   10044/tcp                           dmoj-mathoid-1
3e673dbc8199   ninjaclasher/dmoj-wsevent   "/bin/sh -c 'node /a…"   6 days ago      Up 6 minutes                   15100-15102/tcp                     dmoj-wsevent-1
43e36eb1776d   mariadb                     "docker-entrypoint.s…"   6 days ago      Up 7 minutes                   3306/tcp                            dmoj_mysql
23ef003d5e9e   redis:alpine                "docker-entrypoint.s…"   6 days ago      Up 7 minutes                   6379/tcp                            dmoj_redis

This is the log of the dmoj-bridged container when I run docker logs c0885ad67f79:

 AssertionError
 Traceback (most recent call last):
   File "/site/manage.py", line 14, in <module>
     execute_from_command_line(sys.argv)
   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
     utility.execute()
   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 363, in execute
     settings.INSTALLED_APPS
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 82, in __getattr__
     self._setup(name)
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 69, in _setup
     self._wrapped = Settings(settings_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 170, in __init__
     mod = importlib.import_module(self.SETTINGS_MODULE)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
   File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
   File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
   File "/site/dmoj/__init__.py", line 1, in <module>
     from dmoj.celery import app as celery_app
   File "/site/dmoj/celery.py", line 12, in <module>
     if hasattr(settings, 'CELERY_BROKER_URL_SECRET'):
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 82, in __getattr__
     self._setup(name)
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 69, in _setup
     self._wrapped = Settings(settings_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 170, in __init__
     mod = importlib.import_module(self.SETTINGS_MODULE)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/site/dmoj/settings.py", line 606, in <module>
     assert DMOJ_PDF_PROBLEM_CACHE is None or os.path.isdir(DMOJ_PDF_PROBLEM_CACHE)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 AssertionError

and this is the log of the docker-celery container when I run docker logs 415d28909c93:

 AssertionError

 Usage: celery [OPTIONS] COMMAND [ARGS]...
 Try 'celery --help' for help.

 Error: Invalid value for '-A' / '--app': 
 Unable to load celery application.
 While trying to load the module dmoj_celery the following error occurred:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 57, in convert
     return find_app(value)
            ^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 384, in find_app
     sym = symbol_by_name(app, imp=imp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
     module = imp(module_name, package=package, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 105, in import_from_cwd
     return imp(module, package=package)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
   File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
   File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
   File "/site/dmoj_celery.py", line 12, in <module>
     from dmoj.celery import app  # noqa: E402, F401, imported for side effect
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/site/dmoj/__init__.py", line 1, in <module>
     from dmoj.celery import app as celery_app
   File "/site/dmoj/celery.py", line 12, in <module>
     if hasattr(settings, 'CELERY_BROKER_URL_SECRET'):
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 82, in __getattr__
     self._setup(name)
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 69, in _setup
     self._wrapped = Settings(settings_module)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/django/conf/__init__.py", line 170, in __init__
     mod = importlib.import_module(self.SETTINGS_MODULE)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/site/dmoj/settings.py", line 606, in <module>
     assert DMOJ_PDF_PROBLEM_CACHE is None or os.path.isdir(DMOJ_PDF_PROBLEM_CACHE)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 AssertionError

I hope this helps.

Ninjaclasher commented 1 year ago

Thanks for the bug report. I've pushed a fix (1f8c4c29fdc283a1fbe555ae245550d4b6e67903) that should resolve this. There was an upstream change to verify that the pdfcache is a directory.

Dellos7 commented 1 year ago

Thank you very much! It's working now.

Ninjaclasher / dmoj-docker

Connection error between judge and bridged container #13