search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.28k stars 609 forks source link

[Bug]- Migration from Hive 3.4.4 to Hive 4.1.17 not working #2331

Closed viverma5 closed 2 years ago

viverma5 commented 2 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) RedHat
OS version (client) 8.3.
Virtualized Env. True
Dedicated RAM 16 GB
vCPU 8
TheHive version / git hash 4.1.17
Package Type RPM,
Database Cassandra
Index type Lucene
Attachments storage Local
Browser type & version If applicable

Problem Description

Describe the problem/bug as clearly as possible.

Steps to Reproduce

  1. step 1 Install The Hive 4-4.1.17 version on RHEL machine. Configure database as cassandra, index as lucene and file system is local. Start The Hive service.All goes well
  2. step 2 Now, launch migration to target server with ES 6.8.21 and Hive version 3.4.4. Migration starts but it detects nothing to change and finishes with no change.

`[root@azucl20007 thehive]# /opt/thehive/bin/migrate --main-organisation StSOC --es-uri http://10.28.100.37:9200 --es-index the_hive --exclude-alert-types misp -o /etc/thehive/application.conf [info] 1,666,167/1,769,472KiB(95%) GC:8 (cpu:5% 169ms) [initialisation] [info] Found ElasticSearch 6.8.21 [info] Found Index the_hive_15 [info] Found index with multiple types [info] Initialising cluster [info] Member is Up: akka://TheHiveMigration@127.0.0.1:35141 [info] 1,430,538/1,769,472KiB(81%) GC:9 (cpu:3% 373ms) [initialisation] [info] Full-text index is available (lucene:/opt/thp/thehive/index) single node [info] Creating database schema [info] The field data is indexed [info] 1,458,874/1,769,472KiB(83%) GC:1 (cpu:0% 22ms) [Finalisation] Organisation:1/1(240,525µs) [info] Running check on Tag ... [info] Check on Tag: no change needed [info] Running check on Log ... [info] Check on Log: no change needed [info] Running check on Alert ... [info] Check on Alert: no change needed [info] Running check on Organisation ... [info] Found duplicate entities:

To-om commented 2 years ago

The migration tool found an index with multiple types whereas the index uses a single type. The detection doesn't work correctly (it need a fix). You can force by adding the parameter --es-single-type true.

viverma5 commented 2 years ago

Hello @To-om

I have tested it by adding --es-single-type true in the migration script. This time migration started but I see warning, where one of the customField named "userGroup" in Hive3, is not able to be created in Hive 4 and we get repeated warnings like below-

[warn] Unable to set custom field userGroup="APAC_User" to case #4379: org.thp.scalligraph.NotFoundError: Custom field userGroup not found

This issue is only for this specific customField userGroup. Not sure, what could be the problem.

Besides, once migration was over, I saw that we got 5 ORGANIZATIONS created [2 for admin and 3 for the one which we used as MAIN organization] instead of 2 . Even after re-indexing status is same.

viverma5 commented 2 years ago

Hello @To-om ,

After looking into the logs, I realised error initially while creating customFields in the migration script.

`[error] CustomField creation failure: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array`

I see that CustField is truncated at 7691 characters. I tried to change the data of the CF in Hive 3 to see if it makes any difference, but everytime it fails at same character position 7691 Do we have any kind of restriction? This issue does not happen if I upgrade from 3.4.4 - 4.0.4-X.X.X

To-om commented 2 years ago

Single type detection and custom field truncation problems are fixed in 4.1.18