[Bug] All Orgs disappear after running docker-compose down/up -d

james-drs commented 3 years ago

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian 11
Browser	Chrome, Brave, Safari
Virtualized Env.	False
TheHive version / git hash	4.1.11-1
Package Type	Docker

Problem Description

When standing up a new version of TheHive4 with docker-compose, upon initial sign-in the admin Org is visible. Additional Orgs are visible when created. However, all Orgs disappear after running docker-compose down and docker-compose up -d any time after the initial login.

Since docker-compose down/up -d appears to be required to get TheHive4 to recognize any changes in application.conf, this bug becomes evident when any configuration changes need to be made.

Steps to Reproduce

.env

CORTEX_KEY=[INPUT-CORTEX-KEY-HERE]
JOB_DIRECTORY=/opt/cortex/jobs

docker-compose.yml

version: "3.8"
services:
  nginx:
    container_name: nginx
    hostname: nginx
    image: nginx:1.19.5
    ports:
      - 80:80
      - 443:443
    networks:
      - proxy
    volumes:
      - ./vol/nginx:/etc/nginx/conf.d
      - ./vol/ssl:/etc/ssl
    restart: on-failure

  cassandra:
    container_name: cassandra
    image: cassandra:3.11
    restart: unless-stopped
    hostname: cassandra
    environment:
      - MAX_HEAP_SIZE=1G
      - HEAP_NEWSIZE=1G
      - CASSANDRA_CLUSTER_NAME=thp
    volumes:
      - ./vol/cassandra/data:/var/lib/cassandra/data
    networks:
      - backend

  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:7.11.1
    environment:
      - http.host=0.0.0.0
      - discovery.type=single-node
      - cluster.name=hive
      - script.allowed_types= inline
      - thread_pool.search.queue_size=100000
      - thread_pool.write.queue_size=10000
      - gateway.recover_after_nodes=1
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - ./vol/elasticsearch/data:/usr/share/elasticsearch/data
      - ./vol/elasticsearch/logs:/usr/share/elasticsearch/logs
    networks:
      - backend

  thehive:
    container_name: thehive
    image: 'thehiveproject/thehive4:latest'
    restart: unless-stopped
    depends_on:
      - cassandra
    ports:
      - '0.0.0.0:9000:9000'
    volumes:
      - ./vol/thehive/application.conf:/etc/thehive/application.conf
      - ./vol/thehive/data:/opt/thp/thehive/data
      - ./vol/thehive/index:/opt/thp/thehive/index
    command:
      --cortex-port 9001
      --cortex-keys ${CORTEX_KEY}
    environment:
      - 'CORTEX_KEY=${CORTEX_KEY}'
    networks:
      - proxy
      - backend

  cortex:
    container_name: cortex
    image: thehiveproject/cortex:latest
    depends_on:
      - elasticsearch
    networks:
      - proxy
      - backend
    command:
      --job-directory ${JOB_DIRECTORY}
    environment:
      - 'JOB_DIRECTORY=${JOB_DIRECTORY}'
    volumes:
      - ./cortex/application.conf:/etc/cortex/application.conf
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '${JOB_DIRECTORY}:${JOB_DIRECTORY}'
    ports:
      - '0.0.0.0:9001:9001'
    command:
             --start-docker

networks:
  backend:
  proxy:
    external: false

application.conf

###
## Documentation is available at https://github.com/TheHive-Project/TheHiveDocs/TheHive4
###

## Include Play secret key
# More information on secret key at https://www.playframework.com/documentation/2.8.x/ApplicationSecret
include "/etc/thehive/secret.conf"

# JanusGraph
db {
  provider: janusgraph
  janusgraph {
    storage {
      backend: cql
      hostname: ["cassandra"]

      cql {
        cluster-name: thp       # cluster name
        keyspace: thehive           # name of the keyspace
        read-consistency-level: ONE
        write-consistency-level: ONE
      }
    }

    ## Index configuration
    index {
      search {
        backend: lucene
        directory: /opt/index
      }
    }  
  }
}

storage {
   provider: localfs
   localfs.location: /opt/data
}

play.http.parser.maxDiskBuffer: 50MB

#play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule
cortex {
  servers = [
    {
      name = local
      url = http://URL:9001
      auth {
        type = 'bearer'
        key = "=redacted="
      }
      # HTTP client configuration (SSL and proxy)
      wsConfig {}
      # List TheHive organisation which can use this Cortex server. All ("*") by default
      includedTheHiveOrganisations = ["*"]
      # List TheHive organisation which cannot use this Cortex server. None by default
      excludedTheHiveOrganisations = []
    }
  ]
  # Check job update time intervalcortex
  refreshDelay = 5 seconds
  # Maximum number of successive errors before give up
  maxRetryOnError = 3
  # Check remote Cortex status time interval
  statusCheckInterval = 1 minute
}

# MISP configuration
play.modules.enabled += org.thp.thehive.connector.misp.MispModule
misp {
  interval: 5 min
  servers: [
    {
      name = "MISP THP"            # MISP name
      url = "http://URL" # URL or MISP
      auth {
        type = key
        key = "XXXXXXXXXXXXXXx"             # MISP API key
      }
      wsConfig { ssl { loose { acceptAnyCertificate: true } } }
    }
  ]
}

run docker-compose up -d

Authenticate with default admin creds to TheHive (admin Org will be visible). Create additional Orgs (optional). Set API key in Cortex and change .env & .conf to link Cortex (optional).

run docker-compose down docker-compose up -d

Log into TheHive as admin, Orgs will no longer be visible.

Complementary information

This does appear to be a primarily visual bug as navigating to http://URL:9000/index.html#!/administration/organisations/ORG_NAME/details will provide you with the details of the Org, but no Orgs are visible in the "Organisations" pane.
docker-compose restart and stop/start do not appear to change the behavior in either direction. If the Orgs are visible, they will remain so after the restart/stop/start, if the Orgs are not visible, they remain not visible.
The only way I've found to return the visibility of Orgs is to docker-compose stop thehive cassandra, remove their associated directories in the docker vol directory, then run docker-compose up -d thehive cassandra. This will not work if any settings/data in TheHive/Cassandra are required to be retained as this restores both back to their initial configs.

Any assistance in troubleshooting/resolving this issue would be greatly appreciated!

james-drs commented 3 years ago

This issue also appears to impact the display of Cases.

nadouani commented 3 years ago

This means your cassandra volumes are not persistent.

james-drs commented 3 years ago

Forgive my ignorance, but isn't that what - ./vol/cassandra/data:/var/lib/cassandra/data in the docker-compose is supposed to fix?

Geronaso commented 1 year ago

Any fix to this? I did a Docker compose down and all my data is gone now

muhyuddin commented 10 months ago

Facing same issue, when i restarted host machine. All orgs and all user config is also deleted. Any fixes ?

TheHive-Project / TheHive