search

databricks / spark-xml

XML data source for Spark SQL and DataFrames
Apache License 2.0
500 stars 226 forks source link

when used along with select("*", "_metadata") error #627

Closed writetoarun closed 1 year ago

writetoarun commented 1 year ago

I am getting an error when I try to read along with the metadata of the file using .select("*", "_metadata") [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with the name _metadata cannot be resolved. Did you mean one of the following?

Usage of the select method is described here https://learn.microsoft.com/en-us/azure/databricks/ingestion/file-metadata-column

srowen commented 1 year ago

That doesn't sound related to this project or XML even

writetoarun commented 1 year ago

I am sorry it did not explain correctly in the initial request. I am trying to read multiple XML files into a spark data frame and trying to get the file name of XML into the data frame using the below statement

spark.read.format("com.databricks.spark.xml").option("rowTag","row").load("/raw/Batch/////").select("*", "_metadata")

but get an error _metadata not found