search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.59k stars 3.01k forks source link

502 Bad Gateway #3817

Closed AndrewOkie closed 3 years ago

AndrewOkie commented 3 years ago

My actions before raising this issue

Expected Behaviour

CVAT that works

Current Behaviour

502-s all around. I guess it is something with Traefik not routing to "cvat" container but I'm not sure. I had the same issue a year ago (it was nginx IIRC). Back then I was able to solve the problem by buying the tool from a competitor and forgetting the pain I went through trying to install this tool on my physical machine.

Steps to Reproduce (yes, for bugs)

  1. git clone https://github.com/opencv/cvat
  2. cd cvat
  3. docker-compose up -d
  4. firefox --new-window localhost:8080

Environment

Cloud VM instance with Ubuntu 20.04, Docker CE 20.10. No running containers or other crap that can possibly interfere with CVAT.

Screenshots

Screenshot 2021-10-21 at 00 39 42 Screenshot 2021-10-21 at 00 40 13 Screenshot 2021-10-21 at 00 40 18

Docker Logs

$ docker-compose up Creating network "cvat_cvat" with the default driver Creating traefik ... done Creating cvat_redis ... done Creating cvat_db ... done Creating cvat ... done Creating cvat_ui ... done Attaching to cvat_db, cvat_redis, traefik, cvat, cvat_ui cvat_db | cvat_db | PostgreSQL Database directory appears to contain a database; Skipping initialization cvat_db | cvat_db | 2021-10-20 21:36:22.834 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 cvat_db | 2021-10-20 21:36:22.834 UTC [1] LOG: listening on IPv6 address "::", port 5432 cvat_db | 2021-10-20 21:36:22.836 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" cvat_db | 2021-10-20 21:36:22.858 UTC [21] LOG: database system was interrupted; last known up at 2021-10-20 21:28:42 UTC cvat_db | 2021-10-20 21:36:23.050 UTC [21] LOG: database system was not properly shut down; automatic recovery in progress cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: redo starts at 0/1642488 cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: invalid record length at 0/1642568: wanted 24, got 0 cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: redo done at 0/1642530 cvat_db | 2021-10-20 21:36:23.070 UTC [1] LOG: database system is ready to accept connections cvat_redis | 1:C 20 Oct 21:36:22.823 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo cvat_redis | 1:C 20 Oct 21:36:22.823 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=1, just started cvat_redis | 1:C 20 Oct 21:36:22.823 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf cvat_redis | 1:M 20 Oct 21:36:22.828 Running mode=standalone, port=6379. cvat_redis | 1:M 20 Oct 21:36:22.829 # Server initialized cvat_redis | 1:M 20 Oct 21:36:22.829 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. cvat_redis | 1:M 20 Oct 21:36:22.829 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. cvat_redis | 1:M 20 Oct 21:36:22.829 Ready to accept connections cvat_ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration cvat_ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh traefik | time="2021-10-20T21:36:23Z" level=info msg="Configuration loaded from flags." cvat_ui | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf cvat_ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version cvat | 2021-10-20 21:36:24,621 INFO RPC interface 'supervisor' initialized cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh cvat | 2021-10-20 21:36:24,629 CRIT Server 'unix_http_server' running without any HTTP authentication checking cvat | 2021-10-20 21:36:24,630 INFO supervisord started with pid 1 cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh cvat_ui | /docker-entrypoint.sh: Configuration complete; ready for start up cvat_ui | 2021/10/20 21:36:24 [warn] 1#1: duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98 cvat_ui | nginx: [warn] duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98 cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: using the "epoll" event method cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: nginx/1.21.3 cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424) cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: OS: Linux 5.4.0-77-generic cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: start worker processes cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: start worker process 31 cvat | 2021-10-20 21:36:25,632 INFO spawned: 'ssh-agent' with pid 8 cvat | 2021-10-20 21:36:25,635 INFO spawned: 'clamav_update' with pid 9 cvat | 2021-10-20 21:36:25,655 INFO spawned: 'git_status_updater' with pid 10 cvat | 2021-10-20 21:36:25,658 INFO spawned: 'rqscheduler' with pid 11 cvat | 2021-10-20 21:36:25,665 INFO spawned: 'rqworker_default_0' with pid 12 cvat | 2021-10-20 21:36:25,682 INFO spawned: 'rqworker_default_1' with pid 13 cvat | 2021-10-20 21:36:25,713 INFO spawned: 'rqworker_low' with pid 16 cvat | 2021-10-20 21:36:25,738 INFO spawned: 'runserver' with pid 20 cvat | 2021-10-20 21:36:25,763 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139783414488608 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stdout)> cvat | 2021-10-20 21:36:25,771 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 139783414258176 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stderr)> cvat | 2021-10-20 21:36:25,773 INFO exited: clamav_update (exit status 0; not expected) cvat | 2021-10-20 21:36:25,776 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:36:25,781 DEBG 'ssh-agent' stdout output: cvat | SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK; cvat | echo Agent pid 8; cvat | cvat | 2021-10-20 21:36:25,784 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 3 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:25,789 DEBG 'git_status_updater' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:25,792 DEBG 'rqscheduler' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:25,795 DEBG 'rqworker_default_0' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:25,797 DEBG 'rqworker_default_1' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:25,813 DEBG 'git_status_updater' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:25,822 DEBG 'rqscheduler' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:25,827 DEBG 'rqworker_low' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:25,830 DEBG 'rqworker_default_0' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:25,854 DEBG 'runserver' stderr output: cvat | wait-for-it.sh: waiting for cvat_db:5432 without a timeout cvat | cvat | 2021-10-20 21:36:25,861 DEBG 'rqworker_default_1' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:25,903 DEBG 'rqworker_low' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | cvat | 2021-10-20 21:36:25,905 DEBG 'rqworker_low' stderr output: cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat_db | 2021-10-20 21:36:25.907 UTC [28] LOG: incomplete startup packet cvat | 2021-10-20 21:36:25,922 DEBG 'runserver' stderr output: cvat | wait-for-it.sh: cvat_db:5432 is available after 0 seconds cvat | cvat | 2021-10-20 21:36:25,925 DEBG 'runserver' stderr output: cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | cvat | 2021-10-20 21:36:25,926 DEBG 'runserver' stderr output: cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:26,473 DEBG 'rqscheduler' stderr output: cvat | 21:36:26 Registering birth cvat | cvat | 2021-10-20 21:36:26,522 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,526 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,527 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:36:26,530 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,534 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:36:26,534 DEBG 'rqworker_low' stderr output: cvat | Identity added: /home/django/.ssh/id_rsa (django@3977e1f9cf92) cvat | cvat | 2021-10-20 21:36:26,545 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,552 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:36:26,555 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:36:26,564 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,565 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:36:26,582 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,591 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:36:26,592 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:36:26,602 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,605 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:36:26,637 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,638 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,649 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:36:26,649 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,664 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,665 INFO success: rqscheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,665 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,670 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:36:26,697 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,698 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,708 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:36:26,728 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:36:26,729 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:26,730 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:36:27,733 INFO spawned: 'clamav_update' with pid 89 cvat | 2021-10-20 21:36:27,734 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:27,768 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139783414258128 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stderr)> cvat | 2021-10-20 21:36:27,770 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139783414258320 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stdout)> cvat | 2021-10-20 21:36:28,771 INFO success: clamav_update entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:36:28,772 INFO exited: clamav_update (exit status 0; expected) cvat | 2021-10-20 21:36:28,772 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:36:52,658 DEBG fd 34 closed, stopped monitoring <POutputDispatcher at 139783413801264 for <Subprocess at 139783414089136 with name rqworker_default_1 in state RUNNING> (stderr)> cvat | 2021-10-20 21:36:52,668 DEBG fd 30 closed, stopped monitoring <POutputDispatcher at 139783414089184 for <Subprocess at 139783414089136 with name rqworker_default_1 in state RUNNING> (stdout)> cvat | 2021-10-20 21:36:53,671 INFO exited: rqworker_default_1 (terminated by SIGKILL; not expected) cvat | 2021-10-20 21:36:53,671 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:36:54,681 INFO spawned: 'rqworker_default_1' with pid 117 cvat | 2021-10-20 21:36:54,798 DEBG 'rqworker_default_1' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:36:54,905 DEBG 'rqworker_default_1' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | cvat | 2021-10-20 21:36:54,962 DEBG 'rqworker_default_1' stderr output: cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | cvat | 2021-10-20 21:36:54,965 DEBG 'rqworker_default_1' stderr output: cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:36:55,966 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:37:23,386 DEBG 'runserver' stderr output: cvat | bash: [20: 1 (255)] tcsetattr: Inappropriate ioctl for device cvat | cvat | 2021-10-20 21:37:23,387 DEBG fd 40 closed, stopped monitoring <POutputDispatcher at 139783414089568 for <Subprocess at 139783414089520 with name runserver in state RUNNING> (stdout)> cvat | 2021-10-20 21:37:23,387 DEBG fd 44 closed, stopped monitoring <POutputDispatcher at 139783413802128 for <Subprocess at 139783414089520 with name runserver in state RUNNING> (stderr)> cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) cvat | 2021-10-20 21:37:23,388 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:37:23,401 INFO spawned: 'runserver' with pid 126 cvat | 2021-10-20 21:37:23,525 DEBG 'runserver' stderr output: cvat | wait-for-it.sh: waiting for cvat_db:5432 without a timeout cvat | cvat_db | 2021-10-20 21:37:23.561 UTC [29] LOG: incomplete startup packet cvat | 2021-10-20 21:37:23,586 DEBG 'runserver' stderr output: cvat | wait-for-it.sh: cvat_db:5432 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:37:23,884 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:37:23,890 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:37:23,892 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:37:23,909 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:37:23,915 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:37:24,422 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:37:24,423 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:37:24,442 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:37:24,444 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:37:24,461 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:37:24,473 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:37:48,474 DEBG fd 39 closed, stopped monitoring <POutputDispatcher at 139783413801696 for <Subprocess at 139783414089328 with name rqworker_low in state RUNNING> (stderr)> cvat | 2021-10-20 21:37:48,483 DEBG fd 35 closed, stopped monitoring <POutputDispatcher at 139783414089376 for <Subprocess at 139783414089328 with name rqworker_low in state RUNNING> (stdout)> cvat | 2021-10-20 21:37:49,486 INFO exited: rqworker_low (terminated by SIGKILL; not expected) cvat | 2021-10-20 21:37:49,487 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:37:50,492 INFO spawned: 'rqworker_low' with pid 152 cvat | 2021-10-20 21:37:50,627 DEBG 'rqworker_low' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:37:50,736 DEBG 'rqworker_low' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | cvat | 2021-10-20 21:37:50,759 DEBG 'rqworker_low' stderr output: cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | cvat | 2021-10-20 21:37:50,760 DEBG 'rqworker_low' stderr output: cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:37:51,766 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:38:13,439 DEBG fd 29 closed, stopped monitoring <POutputDispatcher at 139783414259520 for <Subprocess at 139783414089040 with name rqworker_default_0 in state RUNNING> (stderr)> cvat | 2021-10-20 21:38:13,440 DEBG fd 25 closed, stopped monitoring <POutputDispatcher at 139783414089088 for <Subprocess at 139783414089040 with name rqworker_default_0 in state RUNNING> (stdout)> cvat | 2021-10-20 21:38:14,052 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:14,053 INFO exited: rqworker_default_0 (terminated by SIGKILL; not expected) cvat | 2021-10-20 21:38:14,053 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:38:14,065 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:38:14,068 INFO spawned: 'rqworker_default_0' with pid 163 cvat | 2021-10-20 21:38:14,069 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:38:14,104 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:14,105 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:38:14,200 DEBG 'rqworker_default_0' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:38:14,255 DEBG 'rqworker_default_0' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | cvat | 2021-10-20 21:38:14,257 DEBG 'rqworker_default_0' stderr output: cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | cvat | 2021-10-20 21:38:14,257 DEBG 'rqworker_default_0' stderr output: cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:38:15,259 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:38:15,580 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:15,595 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:38:15,597 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:38:15,624 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:15,641 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat | 2021-10-20 21:38:21,296 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 139783414258656 for <Subprocess at 139783414088656 with name git_status_updater in state RUNNING> (stderr)> cvat | 2021-10-20 21:38:21,298 DEBG fd 15 closed, stopped monitoring <POutputDispatcher at 139783414088704 for <Subprocess at 139783414088656 with name git_status_updater in state RUNNING> (stdout)> cvat | 2021-10-20 21:38:22,300 INFO exited: git_status_updater (terminated by SIGKILL; not expected) cvat | 2021-10-20 21:38:22,301 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:38:23,305 INFO spawned: 'git_status_updater' with pid 181 cvat | 2021-10-20 21:38:23,423 DEBG 'git_status_updater' stderr output: cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout cvat | cvat | 2021-10-20 21:38:23,482 DEBG 'git_status_updater' stderr output: cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device cvat | bash: no job control in this shell cvat | cvat | 2021-10-20 21:38:24,486 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) cvat | 2021-10-20 21:38:32,396 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:32,397 INFO reaped unknown pid 149 cvat | 2021-10-20 21:38:32,397 DEBG received SIGCHLD indicating a child quit cvat | 2021-10-20 21:38:32,415 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 17 cvat | cvat | 2021-10-20 21:38:32,416 DEBG 'ssh-agent' stderr output: cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none) cvat | cvat | 2021-10-20 21:38:32,441 DEBG 'ssh-agent' stderr output: cvat | debug2: fd 4 setting O_NONBLOCK cvat | cvat | 2021-10-20 21:38:32,446 DEBG 'ssh-agent' stderr output: cvat | debug1: process_message: socket 1 (fd=4) type 11 cvat | cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:43 +0000] "GET / HTTP/1.1" 200 1045 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1" cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:45 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js HTTP/1.1" 200 5955484 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1" cvat | 2021-10-20 21:38:45,441 INFO reaped unknown pid 151 cvat | 2021-10-20 21:38:45,448 DEBG received SIGCHLD indicating a child quit cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:45 +0000] "GET /favicon.ico HTTP/1.1" 200 102581 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1" cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:51 +0000] "GET /download.worker.cb069004e9ace5f89e8e.js HTTP/1.1" 200 16721 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1" cvat | 2021-10-20 21:39:15,921 INFO reaped unknown pid 150 cvat | 2021-10-20 21:39:15,922 DEBG received SIGCHLD indicating a child quit

bsekachev commented 3 years ago

Hi Andrew,

What is docker ps output?

azhavoro commented 3 years ago

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

AndrewOkie commented 3 years ago

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

Yes, indeed, it is a small machine with 570 MiB of free RAM available. How much does CVAT require to work?

azhavoro commented 3 years ago

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

Yes, indeed, it is a small machine with 570 MiB of free RAM available. How much does CVAT require to work?

If you do not want to use any additional components such as analytics or run automatic annotations, I think 4-8 GB of total RAM will be enough.

azhavoro commented 3 years ago

I hope you solved the issue, don't hesitate to reopen if you still have questions.