search

DFIRKuiper / Kuiper

Digital Forensics Investigation Platform
772 stars 111 forks source link

docker-compose issue to start all images #45

Closed k2kemp closed 2 years ago

k2kemp commented 2 years ago

I installed Kuiper on a new machine with this setting : OS: 64-bit Ubuntu 18.04.6 LTS RAM: 8GB Cores: 4 Disk: 50GB

Docker Engine 20.10.12 docker-compose 1.29.2

Sadly, i tried many times and i always have error and the last one is the all images didn't started correctly : image

Any ideas how to make it works ?

salehmuhaysin commented 2 years ago

hi,

try to change the owner of elasticsearch folder to your user account then try again also please share the logs for es01 service

k2kemp commented 2 years ago

Now after a reboot, i get this error message image

salehmuhaysin commented 2 years ago

make sure the nfs docker running

harrim4n commented 2 years ago

I'm facing the same issue. Kuiper started once, but then the es01 container exited with "Exit 78". Now, it doesn't start at all because of the same permission error (also for celery). There is a similar error mentioned in the Readme (for the flask container), but the suggested solution of re-running docker-compose up does not work.

The error for es01 was only the max_map_count issue, I neglected to set the variable again after a reboot of the host. Maybe the documentation in the Readme should be changed to automatically set this after a reboot instead of just temporarily?

salehmuhaysin commented 2 years ago

hi could you share the logs for the service that failed running

docker-compose logs -f --tail=100 <service>
harrim4n commented 2 years ago

The initial failure was only related to vm.max_map_count, so no further investigation is needed there.

The error with the permissions / connection refused only happens when you run docker-compose stop (or the containers are stopped for another reason) and then run docker-compose up.

Then some variation of the following error is shown:

$ docker-compose start                                                                                   
Starting mongodb ... done                                                                                                      
Starting es01    ... done                                                                                                      
Starting redis   ... done                                                                                                      
Starting nfs     ... done
Starting celery  ... error
Starting flask   ... error
Starting nginx   ... error

ERROR: for celery  Cannot start service celery: error while mounting volume '/var/lib/docker/volumes/kuiper_kuiper_nfs/_data': failed to mount local volume: mount :/:/var/lib/docker/volumes/kuiper_kuiper_nfs/_data, data: addr=172.30.250.10: connection ref
used

docker-compose logs -f --tail=100 celery or similar does not show any (relevant) logs, as the container doesn't start in the first place.

salehmuhaysin commented 2 years ago

does docker-compose down and docker-compose up -d solve the issue? usually the errors "permissions / connection refused" occure when celery or flask container try to connect to nfs contianer to mound the volume and nfs did not came up yet.

cyber1c3 commented 2 years ago

i have the same docker-compose up error: my env info: docker version: 18.09.0 docker-compose version: v2.2.3 i have do sysctl -w vm.max_map_count=262144 everytime i exec docker-compose up -d ,allways the es01 service error

# docker-compose ps -a
NAME                COMMAND                  SERVICE             STATUS              PORTS
kuiper_celery       "/bin/sh -c 'cron &&…"   celery              running             
kuiper_es01         "/bin/tini -- /usr/l…"   es01                exited (1)          
kuiper_flask        "/bin/sh -c 'cron &&…"   flask               running             0.0.0.0:5000->5000/tcp
kuiper_mongodb      "docker-entrypoint.s…"   mongodb             running             0.0.0.0:27017->27017/tcp
kuiper_nfs          "/usr/bin/nfsd.sh"       nfs                 running             0.0.0.0:2049->2049/tcp
kuiper_nginx        "/docker-entrypoint.…"   nginx               running             0.0.0.0:443->443/tcp
kuiper_redis        "docker-entrypoint.s…"   redis               running             0.0.0.0:6379->6379/tcp

it's logs is: docker-compose logs -f --tail=100 es01

kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,448Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [runtime-fields-common]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,448Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [search-business-rules]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,448Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [searchable-snapshots]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,449Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [snapshot-repo-test-kit]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,449Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [spatial]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,449Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [transform]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,449Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [transport-netty4]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,449Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [unsigned-long]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,450Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [vector-tile]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,450Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [vectors]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,450Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [wildcard]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,450Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-aggregate-metric]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,450Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-analytics]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,451Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-async]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,451Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-async-search]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,451Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-autoscaling]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,451Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-ccr]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,452Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-core]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,452Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-data-streams]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,452Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-deprecation]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,452Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-enrich]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,453Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-eql]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,453Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-fleet]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,453Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-graph]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,453Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-identity-provider]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,453Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-ilm]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,454Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-logstash]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,454Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-ml]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,454Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-monitoring]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,454Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-ql]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,455Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-rollup]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,455Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-security]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,455Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-shutdown]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,456Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-sql]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,456Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-stack]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,456Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-text-structure]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,456Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-voting-only-node]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,456Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "loaded module [x-pack-watcher]" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,457Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "no plugins loaded" }
kuiper_es01  | {"type": "server", "timestamp": "2022-02-15T10:38:38,619Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "uncaught exception in thread [main]", 
kuiper_es01  | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
kuiper_es01  | "at org.elasticsearch.node.Node.<init>(Node.java:1090) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "... 6 more",
kuiper_es01  | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
kuiper_es01  | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
kuiper_es01  | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
kuiper_es01  | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
kuiper_es01  | "at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398) ~[?:?]",
kuiper_es01  | "at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]",
kuiper_es01  | "at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]",
kuiper_es01  | "at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]",
kuiper_es01  | "at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.node.Node.<init>(Node.java:427) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]",
kuiper_es01  | "... 6 more"] }
kuiper_es01  | uncaught exception in thread [main]
**kuiper_es01  | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];**
kuiper_es01  | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
kuiper_es01  |  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
kuiper_es01  |  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
kuiper_es01  |  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
kuiper_es01  |  at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398)
kuiper_es01  |  at java.base/java.nio.file.Files.createDirectory(Files.java:700)
kuiper_es01  |  at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
kuiper_es01  |  at java.base/java.nio.file.Files.createDirectories(Files.java:793)
kuiper_es01  |  at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
kuiper_es01  |  at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
kuiper_es01  |  at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
kuiper_es01  |  at org.elasticsearch.node.Node.<init>(Node.java:427)
kuiper_es01  |  at org.elasticsearch.node.Node.<init>(Node.java:309)
kuiper_es01  |  at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
kuiper_es01  |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
kuiper_es01  |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
kuiper_es01  |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
kuiper_es01  |  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
kuiper_es01  |  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
kuiper_es01  |  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
kuiper_es01  |  at org.elasticsearch.cli.Command.main(Command.java:77)
kuiper_es01  |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
kuiper_es01  |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
kuiper_es01  | For complete error details, refer to the log at /usr/share/elasticsearch/logs/es-docker-cluster.log

this is a AccessDeniedException, everytime i try docker-compose down and docker-compose up -d do not solve the issue

i also try chmod -R 755 ./ in Kuiper root dir before i exec docker-compose pull, but it still no work....

so why it always that?

cyber1c3 commented 2 years ago

ohhhhh, i sove this issue, just do chmod -R 777 ./elasticsearch before docker-compose up -d.... so easy!

salehmuhaysin commented 2 years ago

thank you @w1th1c3 The es01 container changes the owner of the ./elasticsearch folder when the container up for the first time. I dont know why, however, to solve the issue change the owner of the folder or change the permissions after first run

chmod <user>:<user> -R ./elasticsearch or chmod -R 777 ./elasticsearch

harrim4n commented 2 years ago

It seems we are experiencing different errors. The chown/chmod for elasticsearch does nothing for me. However, docker-compose down && docker-compose up -d does work around it.