impossible to search job with observable field

[ghislain-bernard]

Impossible to search job with observable field

Request Bug

Work Environment

Question	Answer
OS version (server)	Archlinux
OS version (client)	Archlinux
Cortex version / git hash	3.0.1
Package Type	Docker
Browser type & version	curl

Problem Description

In the job panel, the search of jobs is impossible with observable field not empty

Steps to Reproduce

1. Fill the observable field
2. Click to search
3. Nothing...

Complementary information

This exception appears in the LOGS with the 8.8.8.8 value in observale :

cortex    | [error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_4/_search?
cortex    | StringEntity({"version":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"job"}}},{"bool":{"must":[{"term":{"organization":{"value":"local"}}},{"bool":{"must":[{"term":{"data":{"value":"8.8.8.8"}}},{"bool":{"must_not":[{"term":{"status":{"value":"Deleted"}}}]}}]}}]}}]}},"from":0,"size":10,"sort":[{"createdAt":{"order":"desc"}},{"_id":{"order":"desc"}}]},Some(application/json))
cortex    |  => ElasticError(search_phase_execution_exception,all shards failed,None,None,None,List(ElasticError(query_shard_exception,Binary fields do not support searching,Some(ROiuxZqqStODqmwWsQrMzw),Some(cortex_4),None,null,None)),None)
cortex    | [info] o.t.c.s.ErrorHandler - POST /api/job/_search?range=0-10&sort=-createdAt returned 400
cortex    | org.elastic4play.SearchError: all shards failed
cortex    |     at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:147)
cortex    |     at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
cortex    |     at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
cortex    |     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
cortex    |     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
cortex    |     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
cortex    |     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
cortex    |     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
cortex    |     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
cortex    |     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)

A sample CURL request with the observale field (data) show the problem :

# curl --request GET   --url http://localhost:9200/cortex_4/_search   --header 'content-type: application/json'   --data '{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "data": {
              "value": "8.8.4.4"
            }
          }
        }
      ]
    }
  }
}'
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Binary fields do not support searching","index_uuid":"1FBCuimpRlKwWwTCyOSv7Q","index":"cortex_4"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"cortex_4","node":"1u5gKO9VSAy38DPA4cwRVg","reason":{"type":"query_shard_exception","reason":"Binary fields do not support searching","index_uuid":"1FBCuimpRlKwWwTCyOSv7Q","index":"cortex_4"}}]},"status":400}

Without the field observale (data) the search is OK :

# curl --request GET   --url http://localhost:9200/cortex_4/_search   --header 'content-type: application/json'   --data '{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "relations": {
              "value": "job"
            }
          }
        }
      ]
    }
  }
}'
{"took":35,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":2,"max_score":1.6739764,"hits":[{"_index":"cortex_4","_type":"doc","_id":"U6vJKnABbLIwunFoOeSI","_score":1.6739764,"_source":{"workerId":"fbf736028bb400859eef20f44d96f708","data":"8.8.4.4","dataType":"ip","type":"analyzer","message":"","cacheTag":"6f27554ec11aa04402f2c78070ce471e","createdAt":1581265795420,"createdBy":"ghislain-bernard","organization":"local","tlp":1,"workerDefinitionId":"template_0_0_1","workerName":"template_0_0_1","relations":"job","parameters":"{}","pap":1,"status":"Success","updatedBy":"ghislain-bernard","startDate":1581265796290,"updatedAt":1581265862412,"endDate":1581265862410}},{"_index":"cortex_4","_type":"doc","_id":"VKvJKnABbLIwunFoc-T0","_score":0.2876821,"_source":{"workerId":"fbf736028bb400859eef20f44d96f708","data":"8.8.8.8","dataType":"ip","type":"analyzer","message":"","cacheTag":"46c9360294f3402d9e6788d171f2a6de","createdAt":1581265810387,"createdBy":"ghislain-bernard","organization":"local","tlp":1,"workerDefinitionId":"template_0_0_1","workerName":"template_0_0_1","relations":"job","parameters":"{}","pap":1,"status":"InProgress","updatedBy":"ghislain-bernard","startDate":1581265811270,"updatedAt":1581265811274}}]}}

[amr-cossi]

Same for me on Cortex 3.0.1-1 with an ElasticSearch 5.6.0.

[LetMeR00t]

Hi, Same issue for me, even after deleting and creating the Cortex index in ES.

Cortex: 3.0.1-1 Elastic4Play: 1.11.5 Play: 2.6.23 Elastic4s: 6.5.1 ElasticSearch client: 6.5.2

[grzegorznowacki]

Hello, can we expect the problem to be fixed in the near future?

[mamoedo]

Same problem here.

Cortex: 3.1.1-1
Elastic4Play: 1.13.1
Play: 2.8.3
Elastic4s: 7.9.1
ElasticSearch client: 7.9.1

[keleticsaba]

Is it still a problem?

I still have this issue

Updated today, running: Cortex 3.1.8-1 Elastic4Play 1.13.6 Play 2.8.19 Elastic4s 7.17.2 ElasticSearch client 7.17.1