vpipkt
commented
4 years ago @Atika9 Thanks for filing the issue!
The reading of different resolutions across bands is a known limitation in RasterFrames RasterSource.
We may be able to work around it, and I'd be interested to see if the work around is performant for your use case. Below is a brief snippet of code that does the following to work around:
1) Read the 10m bands, with known tile size and spatial partitioning (available on develop branch but not yet released)
2) Read the 20m bands
- half the 10m tile size
- Upsample by 2
- Use same spatial partitioning as 10m
3)
jointhe two together
When I try this with the URI's in the stack trace you posted it results in an empty DataFrame, because the 10 and 20m bands have different spatial extents. Note band 11's upper coordinate is 10m higher than band 2.
The reading of a catalog does assume that the extents of the products is the same for each band, however I do not think there is anything to explicitly enforce it.
from pyspark.sql.functions import col
df2348 = spark.read.raster(
[[f'https://datarabat.s3.us-east-2.amazonaws.com/b{b}.tif' for b in [2, 3, 4, 8]]],
tile_dimensions=(256, 256),
spatial_index_partitions=200,
).select(
# rename columns
col('proj_raster_0').alias('b2'),
col('proj_raster_1').alias('b3'),
col('proj_raster_2').alias('b4'),
col('proj_raster_3').alias('b8'),
)
df1112 = spark.read.raster(
[[f'https://datarabat.s3.us-east-2.amazonaws.com/b{b}.tif' for b in [11, 12]]],
tile_dimensions=(128, 128),
spatial_index_partitions=200,
).select(
# UPSAMPLE and rename columns
rf_resample('proj_raster_0', lit(2)).alias('b11'),
rf_resample('proj_raster_1', lit(2)).alias('b12'),
)
cond = [
rf_crs(df2348.b2) == rf_crs(df1112.b11),
rf_extent(df2348.b2) == rf_extent(df1112.b11)
]
df = df2348.join(df1112, cond)
df.printSchema()
Returns:
root
|-- b2: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
|-- b3: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
|-- b4: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
|-- b8: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
|-- b11: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
|-- b12: struct (nullable = true)
| |-- tile_context: struct (nullable = true)
| | |-- extent: struct (nullable = false)
| | | |-- xmin: double (nullable = false)
| | | |-- ymin: double (nullable = false)
| | | |-- xmax: double (nullable = false)
| | | |-- ymax: double (nullable = false)
| | |-- crs: struct (nullable = false)
| | | |-- crsProj4: string (nullable = false)
| |-- tile: tile (nullable = false)
metasim
Atika9
When I try to read multiple bands of different sizes I get this error:
Any help is appreciated.