Closed EFT-Defra closed 2 years ago
This did not solve my problem. I'm using CDAP SND.
from pyspark_vector_files import __version__, read_vector_files
assert __version__=='0.2.2'
df = read_vector_files(
path = '/dbfs/mnt/bronze/Ordnance_Survey/Master_Map/Gzip_GML/latest/',
suffix = '.gz',
layer_identifier = 'TopographicLine',
vsi_prefix = '/vsigzip/',
)
Returns Error: PythonException: 'pyarrow.lib.ArrowTypeError: Expected bytes, got a 'list' object'. Full traceback below:
Edit:
The data is not corrupted. I have successfully read this with geopandas.read_file
, saved it df.to_parquet
, and reread it in spark.
It works if you specify the schema and coerce to it:
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, FloatType, ArrayType, BinaryType, BooleanType
from pyspark_vector_files import read_vector_files
schema = StructType([
StructField("fid", StringType(), True),
StructField("featureCode", IntegerType(), True),
StructField("version", IntegerType(), True),
StructField("versionDate", StringType(), True),
StructField("theme", ArrayType(StringType(), True), True),
StructField("accuracyOfPosition", ArrayType(StringType(), True), True),
StructField("changeDate", ArrayType(StringType(), True), True),
StructField("reasonForChange", ArrayType(StringType(), True), True),
StructField("descriptiveGroup", ArrayType(StringType(), True), True),
StructField("descriptiveTerm", ArrayType(StringType(), True), True),
StructField("nonBoundingLine", BooleanType(), True),
StructField("make", StringType(), True),
StructField("physicalLevel", IntegerType(), True),
StructField("physicalPresence", StringType(), True),
StructField("geometry", BinaryType(), True)
])
df = read_vector_files(
path = '/dbfs/mnt/bronze/Ordnance_Survey/Master_Map/Gzip_GML/latest/',
suffix = '.gz',
layer_identifier = 'TopographicLine',
vsi_prefix = '/vsigzip/',
schema=schema,
coerce_to_schema=True,
)
display(df)
Which means it's probably a GML-being-a-list issue.
Specifying the schema isn't very user-friendly though, so I think it's worth exploring other options: #23
Closes #14 by extending
OGR_TO_SPARK
andSPARK_TO_PANDAS
lookups to include OGR subtypes such asOFSTBoolean
andOFSTFloat32
.@aw-west-defra, I think this works but I haven't been able to test it with the dataset that was giving you trouble in the first place, can you double check it for me, please?