Docker starter template

priamai commented 2 years ago

Request Type

Docker examples

Feature Description

Provide better docker composer file for standalone version with volume mounts.

Possible Solutions

This is my starting docker-compose file, however a few questions:

What volumes should I mount to keep persistence? For example where Cortex data resides?
How is S3 used internally? I can see is using a default setting in the log output, is it using Minio internally?
What's the best way to add SSL, documentation talks about reverse proxy but what we should use in Docker (Traefik)?
What's the default user/password? I can see the secret but now the user?
Is this folder still used: /opt/thp/thehive/files ?

version: "3"
services:
  thehive:
    image: strangebee/thehive:latest
    mem_limit: 1500m
    user: root
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - JVM_OPTS="-Xms1024M -Xmx1024M"
    command:
      - --secret
      - "mySecretForTheHive"
    volumes:
      - thehive_db:/data/db

volumes:
  thehive_db:

priamai commented 2 years ago

Would be nice also to have a full configuration example and minimum configuration example for the application.conf. I am assuming that format has changed since version 4 right?

priamai commented 2 years ago

the default user/pass are like previous versions: admin@thehive.local/secret clarity should be done on the secret command parameter which is no the default password but rather the application session secret?

priamai commented 2 years ago

I also noticed that with the standard configuration Cortex seems to be dead?

Icon is red lower left corner.

In fact I can see this error logs:

thehive_1  | [error] o.t.t.c.c.s.AnalyzerSrv [3cd704b8caf2a75a|1e9fa22e27f7effc] List Cortex analyzers fails on cortex0
thehive_1  | java.net.UnknownHostException: cortex: Temporary failure in name resolution
thehive_1  |    at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
thehive_1  |    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
thehive_1  |    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
thehive_1  |    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
thehive_1  |    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
thehive_1  |    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
thehive_1  |    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
thehive_1  |    at play.shaded.ahc.io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:161)
thehive_1  |    at play.shaded.ahc.io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:158)
thehive_1  |    at java.base/java.security.AccessController.doPrivileged(Native Method)

Seems like Cortex is not included in the docker image, I cannot see any logs related to Cortex. What's the desired setup, should we use the old Cortex docker image?

priamai commented 2 years ago

Okay this seems to be a good starting point:

version: "3"
services:
  thehive:
    image: strangebee/thehive:latest
    mem_limit: 1500m
    user: root
    ports:
      - "9000:9000"
    environment:
      - JVM_OPTS="-Xms1024M -Xmx1024M"
    command:
      - --secret
      - "mysecret"
      - "--no-config-cortex"
    volumes:
      - thehive_db:/data/db
      - thehive_files:/data/files
      - thehive_index:/data/index
volumes:
  thehive_db:
  thehive_files:
  thehive_index:

However strange thing is I don't see the application.conf generated in the /data folder of the container.

root@8f1f2df95462:/data# ls -la
total 20
drwxr-xr-x 1 thehive thehive 4096 May 24 13:32 .
drwxr-xr-x 1 root    root    4096 May 24 13:32 ..
drwxr-xr-x 2 root    root    4096 May 24 13:32 db
drwxr-xr-x 2 root    root    4096 May 24 13:32 files
drwxr-xr-x 3 root    root    4096 May 24 13:32 index

This is odd because I didn't provide the argument to skip configuration.

priamai commented 2 years ago

By triggering an exception I discovered that the application file is here instead: /etc/thehive/application.conf

thehive_1 | Kamon couldn't load configuration settings from your *.conf files due to: /etc/thehive/application.conf: java.io.FileNotFoundException: /etc/thehive/application.conf (No such file or directory) at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:190)

priamai commented 2 years ago

This is what works for me now with Cortex. I am not sure what latest version of ES, Cortex would work for. I remember last time I was iterating through versions and 7.11.1 and worked.

version: "3"
services:
  thehive:
    image: strangebee/thehive:latest
    mem_limit: 1500m
    user: root
    ports:
      - "9000:9000"
    environment:
      - JVM_OPTS="-Xms1024M -Xmx1024M"
    command:
      - "--secret"
      - "victoriasecret2022!"
      - "--cortex-hostnames"
      - "cortex"
    volumes:
      - thehive_db:/data/db
      - thehive_files:/data/files
      - thehive_index:/data/index

  elastichivecortex:
    image: 'elasticsearch:7.11.1'
    container_name: eshive
    ports:
      - '0.0.0.0:9200:9200'
    environment:
      - http.host=0.0.0.0
      - http.max_content_length=1000mb
      - discovery.type=single-node
      - cluster.name=hive
      - script.allowed_types= inline
      - thread_pool.search.queue_size=100000
      - thread_pool.write.queue_size=10000
      - gateway.recover_after_nodes=1
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS=-Xms2g -Xmx4g
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - es_hive_data:/usr/share/elasticsearch/data
      - es_hive_logs:/usr/share/elasticsearch/logs

  cortex:
    # this should fix the Analyzers/Responders issues
    # https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md#docker
    image: 'thehiveproject/cortex:latest'
    container_name: cortex
    privileged: true
    environment:
      - start_docker=1
    volumes:
      - ./configs/cortex.conf:/etc/cortex/application.conf
    depends_on:
      - elastichivecortex
    ports:
      - '0.0.0.0:9001:9001'

volumes:
  thehive_db:
  thehive_files:
  thehive_index:
  es_hive_data:
  es_hive_logs:

vdebergue commented 2 years ago

Hello @priamai, To answer some of your questions:

What volumes should I mount to keep persistence? For example where Cortex data resides? This depends on the data storage you are using. We recommend using Cassandra + ES + Minio to store your data so you should mount the volumes corresponding to them. On this page there is an example of a docker compose using those components. If you are using lucene + berkeleydb + filestorage, the path inside TheHive container should be /data/db, /data/index and /data/files => We will update the docs to better indicate that. For cortex, it only stores data in elasticsearch
How is S3 used internally? I can see is using a default setting in the log output, is it using Minio internally? S3 is optional and is used by TheHive to store files (attachments, images ...). One can also use a directory (setting --storage-directory <folder> ). In our docker compose example, a Minio instance is used. When the docker container starts, in the entrypoint it should log what storage the container is using (Using S3... or Using local storage in ...)
What's the best way to add SSL That really depends on how and where your dockers are deployed. If unsure I would recommend adding a simple nginx container.
What's the default user/password? Default user is admin (admin@thehive.local) with password secret
Is this folder still used: /opt/thp/thehive/files ? This folder is used by the debian and rpm packages. It is used if your provide your own config and setting --no-config, or if you don't use the entrypoint
Would be nice also to have a full configuration example and minimum configuration example for the application.conf. I am assuming that format has changed since version 4 right? The debian and rpm packages contain a minimum configuration but we should expand the docker documentation there. The format has mostly stayed the same since version 5. A v4 configuration file should still work with v5 (and relevant docs still apply)
Seems like Cortex is not included in the docker image, I cannot see any logs related to Cortex. Cortex is indeed not included in the TheHive docker image. Some users don't use TheHive with Cortex, others have several Cortex instances linked to one TheHive. TheHive 5 has kept a compatibility with Cortex v3
However strange thing is I don't see the application.conf generated in the /data folder of the container. As this file is generated from the command line arguments, we prefer not to include it in the data folder. However it can be a good start for a custom configuration.
Cortex works with all ES v7 versions. So v7.17.4 should work.

robomotic commented 2 years ago

Thanks @vdebergue for your responses, just more clarity on this:

It is used if your provide your own config and setting --no-config, or if you don't use the entrypoint

In my example case I don't use the --no-config, therefore the application.conf will not be generated anywhere: is this correct?

What would be a way to trigger the creation of that file with standard settings? Should I just use --no-config and --config-file with a path of my choice?

In the documentation it says:

docker run --rm -p 9000:9000 -v <host_data_folder>:/data/files -v <host_conf_folder>:/data/conf <thehive-image> --config-file /data/conf/application.conf

However you said:

As this file is generated from the command line arguments, we prefer not to include it in the data folder. However it can be a good start for a custom configuration.

So does that mean I can choose where to store the configuration file anywhere I wanted?

vdebergue commented 2 years ago

I was not really clear about that:

When you use the docker file, a script is called at startup: the entrypoint /opt/thehive/entrypoint
This script will parse the command arguments, generate a config file and start the application with this config file
By default, the config file is generated in /tmp with a random name. This tmp config file contain the configuration from the arguments and will also include (== import) /etc/thehive/application.conf
--config-file is used when the user wants to give to TheHive a custom configuration. TheHive will not write in this file
When using --no-config, the entrypoint will not generate a configuration file. So a config file will need to be passed using --config-file. So in your case as you don't use it, an application.conf will be generated.

The configuration file is only read and you normally don't need to save it.

To summarize the usage:

if you can customize TheHive with the command arguments, use this method
- don't hesitate to send a feedback if some options are not available and could be added there
If you have further needs (special ssl setup between thehive and db, custom auth, ... ), provide your own application.conf with --no-config and --config-file options

StrangeBeeCorp / TheHive-feedback