taraslayshchuk / es2csv

Export from an Elasticsearch into a CSV file
Apache License 2.0
510 stars 191 forks source link

Can't backup with query #26

Closed tophymastery closed 7 years ago

tophymastery commented 7 years ago

Try to backup csv file with query from json but it show error

Tried with command

es2csv -u http://localhost:9200 -i 'logstash-*' -q @'elk_query.json' -o backup.csv

and here is my elk_query.json

{
   "query" :{
      "range": {
        "@timestamp": {
          "gte": "2017-03-01T20:03:12.000",
          "lte": "2017-03-30T20:03:12.000"
        }
      }
   }
 }

It's show error as below

Traceback (most recent call last):
  File "/usr/local/bin/es2csv", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/es2csv.py", line 279, in main
    es.search_query()
  File "/Library/Python/2.7/site-packages/es2csv.py", line 40, in f_retry
    return f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/es2csv.py", line 134, in search_query
    res = self.es_conn.search(**search_args)
  File "/Library/Python/2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Library/Python/2.7/site-packages/elasticsearch/client/__init__.py", line 569, in search
    doc_type, '_search'), params=params, body=body)
  File "/Library/Python/2.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/Library/Python/2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Library/Python/2.7/site-packages/elasticsearch/connection/base.py", line 122, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'search_phase_execution_exception', u'Failed to parse query [{\n   "query" :{\n      "range": {\n        "@timestamp": {\n          "gte": "2017-03-01T20:03:12.000",\n          "lte": "2017-03-30T20:03:12.000"\n        }\n      }\n   }\n }\n]')

I also try this json query from kibana it's can show result properly, please help. Thank you in advance

taraslayshchuk commented 7 years ago

Hello,

tl;dr version: add -r(--raw_query) argument.

If you would like to use Query DSL syntax of query you should switch query parser to raw mode by adding -r(--raw_query) argument. By default es2csv is using Lucene syntax. Please look into Examples.

taraslayshchuk commented 7 years ago

@tophymastery, does it resolve your issue?