Closed hupiv closed 6 years ago
@hupiv Please keep these sorts of questions on our forums. Github is reserved for confirmed bugs and feature tracking.
In this case you are loading a field which contains multiple values in it, but Spark is treating the field as a singular value. You'll need to use es.read.field.as.array.include
to demarcate which fields have multiple values.
I'm seeing this too. In what sense is it not a bug? The elasticsearch connector isn't properly inferring the schema.
In addition, there's no indication from the error of what field is causing the issue. And if this needs to be set in the configuration loaded on startup, it impedes use of elastic search when using spark with a REPL.
@elbamos
In what sense is it not a bug? The elasticsearch connector isn't properly inferring the schema.
Any field in Elasticsearch can be a bag of values. There's really nothing to go on in terms of what is returned from Elasticsearch's mapping end point that would indicate that a field holds multiple values. We need assistance from the user to determine the fields that are lists of values.
In addition, there's no indication from the error of what field is causing the issue.
This is partly a limitation with how Spark is laid out. As you'll see, none of ES-Hadoop/Spark's packages are present in the stack traces above. Spark is simply being given what it considers conflicting data types. Perhaps we can add a warning to the logs that states that we've found an unmarked array field while working in a context that expects consistent field contents. Please remember that our connector supports many different integrations, some of which are not tied to schemas and allow for much looser definitions of what is allowed in each field of a record.
And if this needs to be set in the configuration loaded on startup, it impedes use of elastic search when using spark with a REPL.
It should be noted here that configurations can be passed to the connector both by the spark session configuration, set at the beginning of the REPL, but also by providing a map of configuration values for each RDD and Dataset at creation time.
Thank you for your feedback. We'll work toward improving the clarity of these sorts of errors soon.
Hello, i've this error while using "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "6.3.0" could you please help me resolve that: scala.MatchError: Buffer(00205215) (of class scala.collection.convert.Wrappers$JListWrapper)
Issue description
I'm trying to read from Elasticsearch 5.6.0 with spark 2.1. I have the error message "scala.MatchError: Buffer(ACK) (of class scala.collection.convert.Wrappers$JListWrapper)"
Steps to reproduce
Code:
Print schema:
Strack trace: