taraslayshchuk / es2csv

Export from an Elasticsearch into a CSV file
Apache License 2.0
510 stars 191 forks source link

Getting "TypeError: unsupported operand type(s) for -: 'dict' and 'int'" #74

Open yahavb opened 4 years ago

yahavb commented 4 years ago

I must overlook something here so sorry for the basic question here!

after installing es2csv I tried to run it against public ES cluster es2csv --verify-certs -u https://myes.com -i logstash-* -D log -q 'spot' -o database.csv

I get

Found {u'relation': u'eq', u'value': 250} results.
Traceback (most recent call last):                                                                                                          
  File "/Users/birayaha/Library/Python/2.7/bin/es2csv", line 11, in <module>
    sys.exit(main())
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/es2csv_cli.py", line 53, in main
    es.search_query()
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/es2csv.py", line 26, in f_retry
    return f(*args, **kwargs)
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/es2csv.py", line 148, in search_query
    bar = progressbar.ProgressBar(widgets=widgets, maxval=self.num_results).start()
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 633, in start
    self.update(self.min_value, force=True)
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 579, in update
    StdRedirectMixin.update(self, value=value)
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 141, in update
    DefaultFdMixin.update(self, value=value)
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 68, in update
    line = converters.to_unicode('\r' + self._format_line())
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 508, in _format_line
    widgets = ''.join(self._to_unicode(self._format_widgets()))
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 473, in _format_widgets
    data = self.data()
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 401, in data
    percentage=self.percentage,
  File "/Users/birayaha/Library/Python/2.7/lib/python/site-packages/progressbar/bar.py", line 320, in percentage
    total = self.max_value - self.min_value
TypeError: unsupported operand type(s) for -: 'dict' and 'int'

Not sure what I need to add to avoid the cast issue.

dkoneill commented 4 years ago

Line 127 in es2csv.py is the cause of the issue and it is because the result set has changed in ElasticSearch 7.0.

self.num_results = res['hits']['total']

returns a dict rather than the number expected. Changing line 127 to the following fixes the issue

self.num_results = res['hits']['total']['value']

This is caused by a breaking change in ElasticSearch 7.0 where hits are now returned as an object. https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#hits-total-now-object-search-response