Search fails when selecting time range

[AbdulRhmanAlfaifi]

Hi :D

Describe the bug When I try selecting a time range I get empty results from the toast notifications. I get this error message in Kuiper.log

"2023-06-21 14:25:40.987342","[ERROR]","elkdb.py.query[Lin.295]","elasticsearch","Query [TEST_CASE] failed [RequestError]","Cannot parse '!(data_type:"tag") AND (machine:"TEST_CASE_TEST_MACHINE" AND Data.@timestamp:[2023\-05\-01T16\:58\:00\ TO\ 2023\-05\-04T16\:58\:00])': Encountered " <RANGE_GOOP> "TO\\ "" at line 1, column 131.
Was expecting:
    "TO" ...
    "

To Reproduce Use the latest commit in this repo (commit id. dcb09077799da9ee9d063f87af8619b7683f0bad)

Expected behavior To retrieve and show records in the time range

Additional context I was able to trace the issue to the function build_search_query in the file browse_artifacts.html https://github.com/DFIRKuiper/Kuiper/blob/745cee7f82961738ef9f76306f2914d5b0847c0d/kuiper/app/templates/case/browse_artifacts.html#L1592

This function removes bad character and prepare ES query. However, it escapes the space characters between the timestamp and the keyword TO which is not a valid ES query.

When I remove the space from this list (i.e ' ') it works as expected. However, any search query with a space will fail

From git blam this line was added 3 years ago. However, the issue is very recent.

I tried this commit 13d7488719b020059f4c22cca9e36336ebfe9cb1 (commit was on Feb this year) and it works without issues but the commit dcb09077799da9ee9d063f87af8619b7683f0bad have this issue. So, the bug was introduced somewhere between these commits

Thanks!

[salehmuhaysin]

could you check the data type of the field, it should be date the escape should only be applied to text fields https://github.com/DFIRKuiper/Kuiper/blob/745cee7f82961738ef9f76306f2914d5b0847c0d/kuiper/app/templates/case/browse_artifacts.html#L1588C46-L1588C59

[AbdulRhmanAlfaifi]

Ok, I will check that next week

Happy Eid 🥳🎉

[AbdulRhmanAlfaifi]

Hala @salehmuhaysin ✋

Yes the type of the field @timestamp is text, here is the command I executed:

curl http://127.0.0.1:9200/test/_mapping -s | jq '.test.mappings.properties.Data.properties."@timestamp"'

output:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  },
  "copy_to": [
    "catch_all"
  ],
  "analyzer": "default"
}

I noticed something, after sometime the field type will change to date. I had to remove the index and all data then parse the artifact again for it to show me the error.

NOTE: I am parsing kjson file for this error to show, However in some cases I noticed the error while parsing registry files (I am not sure which parser is it REGTimeline or AutoParser @muteb & @mayHamad might be able to help). The kjson file has timestamps in the field @timestamp in the format YYYY-MM-DD HH:MM:SS but some of the records contain an empty string (i.e "" not null) in the field @timestamp

[salehmuhaysin]

hello it is data type auto-detect, so it will consider the first records with the field @timestamp and then decide what is the field type, maybe these records that does not include seconds corrupted the field type and make it text. yes if the field parsed as text you need to delete the index to fix the mapping

[AbdulRhmanAlfaifi]

Could we add a check to Kuiper before it pushes the record to the index to check for @timestamp field and if it can't parse it then it should default to 1700-01-01T00:00:00? I see here you already do that for empty fields (i.e null): https://github.com/DFIRKuiper/Kuiper/blob/dcb09077799da9ee9d063f87af8619b7683f0bad/kuiper/app/controllers/parser_management.py#L660

could we add one more check like this:

if '@timestamp' not in data[d]['Data'] or data[d]['Data']['@timestamp'] is None or data[d]['Data']['@timestamp'] is "":

[salehmuhaysin]

done, fixed the issue it will check for both empty string and if date should be ISO format r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z?$'