search

taraslayshchuk / es2csv

Export from an Elasticsearch into a CSV file
Apache License 2.0
510 stars 191 forks source link

File too large json.dumps #36

Closed iyabchen closed 6 years ago

iyabchen commented 6 years ago

Tried to dump an index which is more than 300+ MB, and run into

$ es2csv -q '' -i 'metrics-2017.10.20' -o /tmp/test.csv -u http://172.18.0.2:9200 Found 679860 results Run query [ ] [30701/679860] [ 4%] [0:00:09] [ETA: 0:03:26] [ 3.1 Kidocs/s]Traceback (most recent call last): File "/usr/local/bin/es2csv", line 11, in sys.exit(main()) File "/usr/local/lib/python2.7/dist-packages/es2csv.py", line 283, in main es.search_query() File "/usr/local/lib/python2.7/dist-packages/es2csv.py", line 40, in f_retry return f(args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/es2csv.py", line 170, in search_query self.flush_to_file(hit_list) File "/usr/local/lib/python2.7/dist-packages/es2csv.py", line 212, in flush_to_file tmp_file.write('%s\n' % json.dumps(out)) IOError: [Errno 27] File too large

taraslayshchuk commented 6 years ago

Hello, what filesystem are you using? This error is weird. I have been worked with indexes which is more 1 GB size with out any issues.

iyabchen commented 6 years ago

Cannot reproduce. maybe due to lack of space in the hard drive. Thanks for answering.