search

liquidinvestigations / node

Deploy Liquid Investigations on Nomad
https://github.com/liquidinvestigations/docs/wiki
MIT License
6 stars 6 forks source link

Errors running liquid deploy #343

Closed adieyal closed 1 year ago

adieyal commented 1 year ago

I've followed the instructions at https://github.com/liquidinvestigations/cluster/blob/master/Readme.md and https://github.com/liquidinvestigations/node#installation but am not able to get all components to run.

When I run ./liquid deploy I get the following:

2023-01-16 11:40:41,080    DEBUG + cat /proc/sys/vm/max_map_count
2023-01-16 11:40:41,325     INFO Starting liquid...
2023-01-16 11:40:41,417     INFO Waiting for health checks on ['core']
2023-01-16 11:40:41,419     INFO [ 0.0] core: http PASSING  +0.0s
2023-01-16 11:40:47,433     INFO Checks green ['core'] after 6.02s
2023-01-16 11:40:47,433     INFO Auth authdemo -> http://authdemo.****.org/__auth/callback
2023-01-16 11:40:47,433    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:47,451    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'authdemo', 'http://authdemo.liquid.****.org/__auth/callback']
2023-01-16 11:40:48,078     INFO Auth hoover -> http://hoover.liquid.****.org//__auth/callback
2023-01-16 11:40:48,078    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:48,097    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'hoover', 'http://hoover.liquid.****.org/__auth/callback']
2023-01-16 11:40:48,686     INFO Auth dokuwiki -> http://dokuwiki.liquid.****.org/__auth/callback
2023-01-16 11:40:48,686    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:48,704    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'dokuwiki', 'http://dokuwiki.liquid.****.org/__auth/callback']
2023-01-16 11:40:49,261     INFO Auth rocketchat-authproxy -> http://rocketchat.****.org/__auth/callback
2023-01-16 11:40:49,261    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:49,280    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'rocketchat-authproxy', 'http://rocketchat.liquid.****.org/__auth/callback']
2023-01-16 11:40:49,846     INFO Auth rocketchat-app -> http://rocketchat.****.org/_oauth/liquid
2023-01-16 11:40:49,846    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:49,861    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'rocketchat-app', 'http://rocketchat.liquid.****.org/_oauth/liquid']
2023-01-16 11:40:50,537     INFO Auth nextcloud -> http://nextcloud.****.org/__auth/callback
2023-01-16 11:40:50,538    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:50,554    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'nextcloud', 'http://nextcloud.liquid.****.org/__auth/callback']
2023-01-16 11:40:51,288     INFO Auth codimd-app -> http://codimd.****.org/auth/oauth2/callback
2023-01-16 11:40:51,289    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:51,308    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'codimd-app', 'http://codimd.liquid.****.org/auth/oauth2/callback']
2023-01-16 11:40:52,010     INFO Auth codimd-authproxy -> http://codimd.****.org/__auth/callback
2023-01-16 11:40:52,010    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:52,030    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'codimd-authproxy', 'http://codimd.liquid.****.org/__auth/callback']
2023-01-16 11:40:52,570     INFO Auth hypothesis -> http://hypothesis.****.org/__auth/callback
2023-01-16 11:40:52,570    DEBUG + docker ps -q -f label=liquid_task=liquid-core
2023-01-16 11:40:52,587    DEBUG + ['docker', 'exec', '156bf0654509', './manage.py', 'createoauth2app', 'hypothesis', 'http://hypothesis.liquid.****.org/__auth/callback']
2023-01-16 11:40:53,153     INFO jobs to stop: set()
2023-01-16 11:40:53,169     INFO Starting hoover-deps...
2023-01-16 11:40:53,264  WARNING service hoover-rabbitmq has no health checks
2023-01-16 11:40:53,264  WARNING service hoover-rabbitmq-prom has no health checks
2023-01-16 11:40:53,264     INFO Waiting for health checks on ['hoover-es-data', 'hoover-es-data-transport', 'hoover-es-master', 'hoover-es-master-transport', 'hoover-rabbitmq-http', 'hoover-tika', 'search-pg', 'snoop-pg']
2023-01-16 11:40:53,272  WARNING [ 0.0]                   snoop-pg: tcp       MISSING  +0.0s
2023-01-16 11:40:53,272  WARNING [ 0.0] hoover-es-master-transport: transport MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]           hoover-es-master: http      MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]       hoover-rabbitmq-http: http      MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]                hoover-tika: http      MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]                  search-pg: tcp       MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]   hoover-es-data-transport: transport MISSING  +0.0s
2023-01-16 11:40:53,273  WARNING [ 0.0]             hoover-es-data: http      MISSING  +0.0s
2023-01-16 11:51:54,147    ERROR [660.9]                   snoop-pg: tcp       MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9] hoover-es-master-transport: transport MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]           hoover-es-master: http      MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]       hoover-rabbitmq-http: http      MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]                hoover-tika: http      MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]                  search-pg: tcp       MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]   hoover-es-data-transport: transport MISSING  +660.9s
2023-01-16 11:51:54,148    ERROR [660.9]             hoover-es-data: http      MISSING  +660.9s
2023-01-16 11:51:54,150    ERROR No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found
2023-01-16 11:51:54,150  WARNING #1/4 retrying in 5 sec
2023-01-16 11:52:04,158    ERROR No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found
2023-01-16 11:51:54,150  WARNING #1/4 retrying in 5 sec
2023-01-16 11:52:04,158    ERROR No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found
2023-01-16 11:52:04,158  WARNING #2/4 retrying in 10 sec
2023-01-16 11:52:24,180    ERROR No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found
2023-01-16 11:52:24,180  WARNING #3/4 retrying in 20 sec
2023-01-16 11:53:04,222    ERROR No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found
Traceback (most recent call last):
  File "./liquid.py", line 70, in <module>
    main()
  File "./liquid.py", line 53, in main
    options.cmd(*extra_args)
  File "/opt/node/liquid_node/commands.py", line 327, in deploy
    retry()(docker.exec_)(f'hoover-deps:search-pg', 'sh', '/local/set_pg_password.sh')
  File "/opt/node/liquid_node/util.py", line 47, in wrapper
    return f(*args, **kwargs)
  File "/opt/node/liquid_node/docker.py", line 59, in exec_
    run_fg(self.exec_command(name, *args), shell=False)
  File "/opt/node/liquid_node/docker.py", line 33, in exec_command
    alloc_id = first(list(allocs()), f'{name} allocs')
  File "/opt/node/liquid_node/util.py", line 9, in first
    assert items, f"No {name_plural} found"
AssertionError: No hoover-deps:search-pg allocs found

docker ps

CONTAINER ID   IMAGE                                         COMMAND                  CREATED          STATUS          PORTS                                                                                                                                     NAMES
d7f710ecdb41   liquidinvestigations/liquid-nextcloud:0.2.3   "/entrypoint.sh /bin�"   28 minutes ago   Up 28 minutes   10.66.60.1:20201->80/tcp, 10.66.60.1:20201->80/udp                                                                                        nextcloud-ce8b3c3e-15a5-ec96-b30d-958fc811f5f6
0989fb2943ed   mariadb:10.4                                  "docker-entrypoint.s�"   28 minutes ago   Up 28 minutes   10.66.60.1:8767->3306/tcp, 10.66.60.1:8767->3306/udp                                                                                      maria-f11bd111-a71a-e143-ad8f-2b0737fa2ca3
66b22658d58f   liquidinvestigations/authproxy:0.3.5          "/bin/sh -c ./authpr�"   28 minutes ago   Up 28 minutes   10.66.60.1:26690->5000/tcp, 10.66.60.1:26690->5000/udp                                                                                    web-019ea714-5e10-bebe-9d99-2cb5f2623d9d
1cba0d2cd53f   liquidinvestigations/authproxy:0.3.5          "/bin/sh -c ./authpr�"   28 minutes ago   Up 28 minutes   10.66.60.1:25836->5000/tcp, 10.66.60.1:25836->5000/udp                                                                                    web-81b587e2-d85d-381e-a4a8-52fc3c98cd00
ee7f188a6a22   liquidinvestigations/authproxy:0.3.5          "/bin/sh -c ./authpr�"   28 minutes ago   Up 28 minutes   10.66.60.1:25272->5000/tcp, 10.66.60.1:25272->5000/udp                                                                                    web-ab2b8d7d-7fa8-c320-3dec-2ac21cf0a47c
6f0b1d5abc22   liquidinvestigations/authproxy:0.3.5          "/bin/sh -c ./authpr�"   28 minutes ago   Up 28 minutes   10.66.60.1:30203->5000/tcp, 10.66.60.1:30203->5000/udp                                                                                    web-910834ae-2d03-bd18-11ad-79e1b631f9dc
4516613c4ff0   liquidinvestigations/liquid-dokuwiki:0.0.3    "/usr/local/bin/tini�"   28 minutes ago   Up 28 minutes   443/tcp, 10.66.60.1:29083->80/tcp, 10.66.60.1:29083->80/udp                                                                               php-b51ed324-ed50-d16a-3216-9a3e8ee44187
156bf0654509   liquidinvestigations/core:0.3.5               "/bin/sh -c ./docker�"   28 minutes ago   Up 28 minutes   10.66.60.1:8768->8000/tcp, 10.66.60.1:8768->8000/udp                                                                                      core-f10ca170-76cd-571b-c324-9bdcf4d01e4b
f5dba7981035   liquidinvestigations/cluster                  "/app/docker-entrypo�"   29 minutes ago   Up 29 minutes                                                                                                                                             cluster

Any help would be appreciated

gabriel-v commented 1 year ago

Hi - sorry for the delay.

That looks like a Nomad error - containers are not started. Look into the Nomad UI at http://10.66.60.1:4646, click around the job "hoover-deps" as it's mentioned in your error message, and report the errors you see there.

Please try again:

Also check out the system requirements - these are checked before deployment, and also by running ./liquid resources.

Another common problem is disk space: ./liquid deploy will download 25GB of docker images to /var/lib/docker, and Nomad will cache those itself, using another 25GB in /opt/cluster/var/.

adieyal commented 1 year ago

thanks - I'll have a look

gabriel-v commented 1 year ago

please reopen if problem persists with latest release in the changelog