How do I add multiple CSV files?

[shyamalee93]

I want to add multiple CSV files as different indices. How do I do it if all the csv files are present in the same /root/csv folder path? How do I add multiple CSV files to the same index as different types?

[flicker581]

I think you need to setup multiple CSV river instances with different indexes, types and filename patterns (filename_pattern).

[shyamalee93]

Oh, okay. I won't be able to add different CSV files as different types for a given index?

[flicker581]

You can do it. Simply create two rivers pointing to the same index. Two rivers cannot point to the same files, but having identical types or pointing to the same index (you case) is allowed.

[shyamalee93]

That is great, okay. When I try to creat a single index with the following code :

curl -XPUT localhost:9200/_river/my_csv_river/_meta -d ' { "type" : "csv", "csv_file" : { "folder" : "/root/csv", "filename_pattern" : ".*.csv$", "fields" : [ "Title", "Tracking", "Timeline", "% Awareness", “Timeline_Num” ], "first_line_is_header" : "true", "field_separator" : ",", "field_id" : "Timeline", }, "index" : { "index" : "awareness", "type" : "csv_type", "bulk_size" : 100, "bulk_threshold" : 10 } }'

I get the following error :

{"error":"MapperParsingException[failed to parse]; nested: JsonParseException[Invalid UTF-8 start byte 0x80\n at [Source: [B@193ad24; line: 12, column: 7]]; ","status":400}

I wanted to try creating two rivers pointing to the same index like you had mentioned. I'm sorry, I'm new to Elasticsearch and programming as such so any help would be appreciated. I keep getting the parsing exception, failed to parse error whenever I change a few properties/ add or remove lines in the cURL request.

[shyamalee93]

• "filename_pattern" : ".*.csv$" That was an error on my part. Now, when I try the index is created but none of the csv values are loaded onto ES. Any ideas? I have given the filename_pattern with two backslashes. Gets uploaded as a comment with only one backslash.

[vtajzich]

if you want to import CSV files you don't have to specify filename_pattern at all. When you want to specify it you have to escape \, see readme where an example is shown.

[vtajzich]

@shyamalee93 could you, please, share some of the files you're trying to upload to ES? as well as the both requests?

[vtajzich]

@shyamalee93 any update?