Autodetect_column_names take header from second row

[lyeesim988]

autodetect_column_name for csv plugin not able to take first row of csv file as header, instead it takes the second row of data as the header. The first run it managed to get the first row as header but if continue with another few times of re-run, the second row of data values are taken as header.

• Version: Logstash 6.2.3

• Operating System: CentOs

• Config File: input { file { path => "/home/data/file.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," autodetect_column_names => "true" }

• Sample Data: "manhatten" : "california" desired output: "city": " california"

• Steps to Reproduce:

  1. Run logstash from command line: bin/logstash -f /home/user/logstash_file.config --path.data /home/user/logstash-6.2.3/var/log/file
  2. First time the header is correctly taken from first row in csv file, which is desired result.
  3. For another few runs, the header is then taken from the data row (second row) which is the row value as header fields.

Similar issues: https://discuss.elastic.co/t/autodetect-columns-from-header-row-csv-ingest/92861 https://discuss.elastic.co/t/autodetect-column-is-taking-data-row-as-column-name-instead-of-header/126383

[siben168]

i have the same issue, I was debugging my conf, but in many case it will load the second row as the header. it is weird that if i add "convert" function, it alwasy messed up headers.

[siben168]

I've learned this from another thread, this is because the race issue for multiple workers. currently the workaround is to set "pipeline.workers: 1" in logstash.yml, it works for me.

It seems that when you have multiple workers, the order of lines could not be guaranteed, definitely this is a bug.

[lyeesim988]

Hi siben168,

Thank you very much. By setting the worker value to 1 has solved my issue.

[pemontto]

Duplicate of #65

[jplew]

in case you don't have a logstash.yml, create a new one in settings/logstash.yml and paste this inside: pipeline.workers: 1

Reference: https://www.elastic.co/guide/en/logstash/6.3/logstash-settings-file.html

[guyboertje]

Duplicate, closing.