TheHive 4 Regularly Crashing - Java Heap Space[Bug]

[stacsirt]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian 10
Virtualized Env.	True
Dedicated RAM	4.04 GB
TheHive version	4.1.14-1
Package Type	DEB
Database	Cassandra
Index type	Elasticsearch

Problem Description

Hi,

Ever since we recently upgraded TheHive to the most recent version (4.1.14-1) along with upgrading Cortex (4) and Elasticsearch (7.15), we've come across issues whereby thehive seems to crash on us constantly, sometimes multiple times during the day and sometimes less. I've attached a screen grab of thehive's logs at the time of a crash - it appears it could be to do with theJava Heap Space:

Unfortunately we have tried multiple things to attempt to fix this (increase heap space, flush cache etc) but no avail. Has anyone else encountered this issue and come up with a solution? I have a feeling it has something to do with Cassandra which we run on the back end.

[dominiksr]

I suggest you go back to version 7.14.x of elasticsearch. I had a big problem using later versions of elasticsearch and had to go back to an earlier one. In a configuration that used docker I could not get elasticsearch indexing to work at all, I analyzed the errors but could not solve this problem, the authors did not answer any of my questions about this. I also do not recommend using a newer version of cassandra than 3.11.11

[dominiksr]

I think you may not have enough ram. You should definitely check the default version of java as well. Cassandra 3.11 supports java 8 so it would be good to set it as default. "sudo update-alternatives --config java"

[stacsirt]

I think you may not have enough ram. You should definitely check the default version of java as well. Cassandra 3.11 supports java 8 so it would be good to set it as default. "sudo update-alternatives --config java"

Thanks for your suggestions! In our case, Elasticsearch indexing seems to work fine, and in general everything works as anticipated until it crashes after an hour or two. With regards to our RAM, we have 4GB RAM on our box. To begin with (right after reboot), it appears to only be taking up to 700MB, but this slowly increases as time goes on until it hits a threshold and crashes the box. We have OpenJDK 11 set as default.

[stacsirt]

If its any help, having analysed our thehive logs further, it appears that before crashing, it looks like it seems to be erroring massively with regards to observable datatypes - we have data flowing in from MISP connections, which might be the source of this, but can't be sure if this is the primary reason (just as a side note, the full error reads: "Observable ** doesn't comply with its schema, field dataType is missing:"):

[GreyGreyman]

@stacsirt In the end, were you able to solve the issue?

[mmbdv]

@GreyGreyman @stacsirt Have you found any solution to fix this problem?

[dominiksr]

You should definitely add more ram and increase the partition/swap file. I increased the swap to double the ram size