search

elephant-track / elephant-server

A server implementation of ELEPHANT
BSD 2-Clause "Simplified" License
7 stars 5 forks source link

(111: Connection refused) while connecting to upstream, client: 127.0.0.1, server #23

Open orena1 opened 1 year ago

orena1 commented 1 year ago

Hi, I've followed the instructions here https://elephant-track.github.io/#/?id=setting-up-with-singularity

git clone https://github.com/elephant-track/elephant-server
cd elephant-server
make singularity-build
make singularity-launch

But I get this error:

base) oamsalem@colab00:~/Downloads/elephant-server$ make singularity-launch
singularity instance start --nv --bind /home/oamsalem/.elephant_binds/var/lib:/var/lib,/home/oamsalem/.elephant_binds/var/log:/var/log,/home/oamsalem/.elephant_binds/var/run:/var/run,/home/oamsalem/Downloads/elephant-server/workspace:/workspace elephant.sif elephant
INFO:    instance started successfully
if [ all = all ]; then \
        singularity exec instance://elephant /start.sh; \
else \
        SINGULARITYENV_CUDA_VISIBLE_DEVICES=all singularity exec instance://elephant /start.sh; \
fi 

Checking for script in /app/prestart.sh
Running script /app/prestart.sh
set RUN_ON_FLASK
2023-08-17 15:43:46,197 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Unlinking stale socket /var/run/supervisor.sock
2023-08-17 15:43:46,521 INFO RPC interface 'supervisor' initialized
2023-08-17 15:43:46,521 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-08-17 15:43:46,521 INFO supervisord started with pid 19
2023-08-17 15:43:47,525 INFO spawned: 'rabbitmq-server' with pid 31
2023-08-17 15:43:47,528 INFO spawned: 'redis-server' with pid 32
2023-08-17 15:43:47,531 INFO spawned: 'uwsgi' with pid 33
2023-08-17 15:43:47,534 INFO spawned: 'celery' with pid 34
2023-08-17 15:43:47,536 INFO spawned: 'nginx' with pid 36
32:C 17 Aug 15:43:47.724 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
32:C 17 Aug 15:43:47.724 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=32, just started
32:C 17 Aug 15:43:47.724 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
32:M 17 Aug 15:43:47.725 * Increased maximum number of open files to 10032 (it was originally set to 1024).
32:M 17 Aug 15:43:47.726 * Running mode=standalone, port=6379.
32:M 17 Aug 15:43:47.726 # Server initialized
32:M 17 Aug 15:43:47.726 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
32:M 17 Aug 15:43:47.726 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
32:M 17 Aug 15:43:47.726 * Ready to accept connections
2023/08/17 15:43:47 [warn] 36#36: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:48 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:48,198 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:48 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:48,740 INFO success: rabbitmq-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,740 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,741 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,741 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:49,744 INFO spawned: 'uwsgi' with pid 263
2023/08/17 15:43:49 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:49 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:49,757 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:50 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:51 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:52.551472-04:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-08-17 15:43:52.562684-04:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-08-17 15:43:52,565 INFO spawned: 'uwsgi' with pid 264
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:52 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:52,581 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:52 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"

  ##  ##      RabbitMQ 3.12.3
  ##  ##
  ##########  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Erlang:      25.3.2.5 [jit]
  TLS Library: OpenSSL - OpenSSL 1.1.1  11 Sep 2018
  Release series support status: supported

  Doc guides:  https://rabbitmq.com/documentation.html
  Support:     https://rabbitmq.com/contact.html
  Tutorials:   https://rabbitmq.com/getstarted.html
  Monitoring:  https://rabbitmq.com/monitoring.html

  Logs: /var/log/rabbitmq/rabbit@colab00.log
        <stdout>

  Config file(s): (none)

  Starting broker...2023/08/17 15:43:53 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:54 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:55,758 INFO spawned: 'uwsgi' with pid 265
2023/08/17 15:43:55 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:55 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:55,778 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:56 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:56,759 INFO gave up: uwsgi entered FATAL state, too many start retries too quickly
 completed with 0 plugins.
2023/08/17 15:43:57 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"

 -------------- celery@colab00 v5.0.5 (singularity)
--- ***** ----- 
-- ******* ---- Linux-5.15.0-46-generic-x86_64-with-debian-buster-sid 2023-08-17 15:43:58
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         main:0x7f66f63a4090
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 40 (thread)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

2023/08/17 15:43:58 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:59 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:00 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:01 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:02 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:03 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
^C2023-08-17 15:44:03,921 WARN received SIGINT indicating exit request

Any suggestions? Thanks, Oren

mestia commented 1 year ago

I could work around this problem by using network instead of a unix socket:

docker/nginx.conf
uwsgi_pass  127.0.0.1:10002;

docker/uwsgi.ini
socket = 127.0.0.1:10002
orena1 commented 1 year ago

Thanks @mestia so what should I do?

mestia commented 1 year ago

You need to clone the repository, modify the 2 mentioned above files and re-build the singularity image with make singularity-build After that one should be able to start the singularity instance and access the server from a browser.