miku / esbulk

Bulk indexing command line tool for elasticsearch.
GNU General Public License v3.0
281 stars 40 forks source link

Support ES 7.x-style mappings with no document type #32

Closed bnewbold closed 3 years ago

bnewbold commented 4 years ago

When trying to index documents into an Elasticsearch 6.x index with no document type (eg, created with include_type_name=false and mappings is directly the schema, not a JSON object with key _doc), I got the following esbulk error:

2020/05/20 13:46:39 using 1 servers
2020/05/20 13:46:39 {[http://localhost:9200] dev_scholar_fulltext default 200 true key http  }
panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 1 [running]:
main.main()
        /home/tir/go/src/github.com/miku/esbulk/cmd/esbulk/esbulk.go:189 +0x1aed

This might be an issue with using ES 6.x in this way, but from the panic text I suspect this is due to esbulk not supporting this newer single-document-type mapping format.

miku commented 4 years ago

Thanks, I'll look into it!

zazi commented 3 years ago

bump. "document type" has been removed in Elasticsearch 7.x, see Removal of mapping types. Using esbulk with a (provided) mapping without "document type" causes an error message á la

failed to apply mapping with 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."}],"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."},"status":400}
miku commented 3 years ago

Yes, that issue has been dangling (along with some better testing setup for various es versions).

miku commented 3 years ago

As a first measure, in 0.7.0 the docType is set to the empty string and will not be included in the bulk request, if not provided.

miku commented 3 years ago

Ok, thanks for all the patience - the issue should be gone with 0.7.1.