file -> elasticsearch: Caught exception in Main process: [object Object]

[jze]

I successfully exported my index from Elasticsearch 1.7.3 to files with the newest Git version (4a4cc8a). However, the re-import does not work. For testing purposes I exported the output of my Marvel plugin. This is the complete log:

nodejs exporter.js -s file -th 172.17.0.1 -ti .marvel-kibana -sf /data/ -si .marvel-kibana -v
{ 'drivers.source': 'file',
  'drivers.target': 'elasticsearch',
  'drivers.dir': [ './drivers' ],
  'run.step': 100,
  'run.concurrency': 4,
  'run.mapping': true,
  'run.data': true,
  'memory.limit': 0.9,
  'errors.retry': 3,
  'log.debug': true,
  'log.enabled': true }
Elasticsearch Exporter - Version 2.0.0
Arguments: [ '-s',
  'file',
  '-th',
  '172.17.0.1',
  '-ti',
  '.marvel-kibana',
  '-sf',
  '/data/',
  '-si',
  '.marvel-kibana',
  '-v' ]
Reading options
Successfully loaded [BigQuery Driver] version: 0.0
Successfully loaded [CSV Driver] version: 0.0
Successfully loaded [Datastore Driver] version: 0.0
Successfully loaded [ElasticSearch Query Driver] version: 0.0
Successfully loaded [ElasticSearch Scroll Driver] version: 1.0
Successfully loaded [Multi File Driver] version: 1.0
Successfully loaded [HBase Driver] version: 0.0
There was an error loading drivers from /usr/share/Elasticsearch-Exporter/drivers
Passing options to drivers for verification
Multi File Driver is verifying options
ElasticSearch Scroll Driver is verifying options
Resetting source driver to begin operations
Passing options to drivers for verification
Multi File Driver is verifying options
ElasticSearch Scroll Driver is verifying options
Fetching source statistics before starting run
Fetching target statistics before starting run
Checking source database health
Fetching mapping from source database
Reading settings for index [.marvel-kibana] type [null]
Reading mapping for index [.marvel-kibana] type [appdata]
ElasticSearch target version:  1.7.3
Checking target database health
Creating indices in target ElasticSearch instance
Created index .marvel-kibana on target ElasticSearch instance
Creating types in target ElasticSearch instance
Created type appdata in target ElasticSearch instance on existing index .marvel-kibana
Mapping on target database is now ready
Concurrency has been disabled because at least one of the drivers doesn't support it
Starting data export
Caught exception in Main process: [object Object]
Exit code 2: Uncaught Exception
Fetched Entries:    0 documents
Processed Entries:  0 documents
Source DB Size:     0 documents
{ source: 
   { version: '1.0',
     status: 'Red',
     docs: { total: 1 },
     aliases: {},
     cluster_status: 'green' },
  target: 
   { version: '1.7.3',
     status: 'yellow',
     indices: [ 'des', '.marvel-kibana' ],
     aliases: {} },
  hits: {},
  memory: { peak: 0, ratio: 0 } }

[jze]

The exported index is so small, I can even attach it here. If you unpack the data with base64 -d | tar xvfz - it will create a directory .marvel-index/ that contains the exported files.

H4sIALQGKlYAA+2WXW/aMBSGueZXRL6mzI4/Qnu7aVeVpmmadhmZ4LXpII4cp9uE8t93EmCIAI0q
pWFrzyNBnOMDPuG8r810pd2jWV79SOc60+9GLwEFIimbK9C+NmMWKhZypYQUIwojuATyRappURZe
uyAYOWv9U3ld8/8p08P+J7bMen/MusFKiCf6z1r954KpUUD7LuQUb7z/7NIFIBel5X+d5wvtez4H
nr//80gp3P+H4Ez/ez0Hnr//S8op7v9DgPv/2+aM/+u33tbo8r9SvOV/ocII/T8E4zWJ02xhfpEb
cqgFMiGx/50bmNiqoo6kC7jf5H3KfVGHisQ6yKL10JYugfGawK/qywJyvUv1EtIejStSm0GETfmU
Xc2jUM5mM5hxJrfOkxvvSlNVWFBXQb32/4z/VzBIs7t+1ujyfySj9vnP4e8i+n8AxutxEJDc2dw4
nxrQY1BHILYT3fYeIhutB2Ru7dLojDTxarJJ38n5OL0AgWd3h9l/td6RPq5fFRbZV5FH/W/5vzDe
w2eKXjXW4X/Gedv/ggmO/h+CjWp3bd9rdnvi7UWVOKM9CC2G86FRFxNCMhqFLIw4JZNdXlau5sbF
9ntc3Gu3qL+SyP10WdbnI6w4u33gX03y+eO327z4snr/4T6WP/d5x7Le1WCazzMqqLy+JtvJ6sT6
YLtlmuimAnbCqvjQr/+hL20vBEEQBEEQBEEQBPkn+ANm04eLACgAAA==

[mallocator]

Thanks for reporting the issue. I'll try to look into it this weekend.

[mallocator]

I took a quick look at the file. It looks like the mapping for appdata is repeated twice. Try doing the import again after fixing .marvel-kibana/appdata/mapping Same goes for the .marvel-kibana/settings file.

Of course this is still a bug that needs to be fixed ^^

[jze]

I have tried to isolate the fault. It seems that the JSON parser is confused if the buffer in driver/file.js near line 271 starts with a line break '\n'. If you add a linelog.error(e); the the function handleUncaughtExceptions in exporter.js you can see this error:

{ name: 'SyntaxError',
  message: 'Unexpected \'\'',
  at: 2,
  text: '\n' }