Closed dolfinus closed 10 months ago
You did not include the column _corrupt_record
in your schema. It's automatically added if you infer the schema, otherwise you need to add it. If not present, it can't be added.
Tried:
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, TimestampType, StringType
spark = SparkSession.builder.config("spark.jars.packages", "com.databricks:spark-xml_2.12:0.17.0").getOrCreate()
schema = StructType([StructField("created-at", TimestampType()), StructField("_corrupt_record", StringType())])
spark.read.format("xml").options(rowTag='item').schema(schema).load("1.xml").show(10, False)
|created-at|_corrupt_record |
|----------|---------------------------------------------------------------------|
|null |<item>\n <created-at>2021-01-01 01:01:01+00:00</created-at>\n</item>|
It is worth mentioning in Readme that _corrupt_record
should be explicitly added to dataframe schema.
Hi.
I'm trying to parse simple xml file:
But if timestamp does not match format, e.g.
T
is replaced with space:null
:I see that there is an option
mode
withPERMISSIVE
as default, which leads towhen it encounters a field of the wrong datatype, it sets the offending field to null
. But malformed value is not being added to column_corrupt_record
because there is nothing wrong with xml structure. So there is no way to detect if input file contains tag with wrong field value ornullValue
, unless user set a differentmode
. Is that desired behavior?