Closed aazuspan closed 2 years ago
Thanks for reporting @aazuspan. I haven't looked into this yet, but I think there's enough information here to figure it out.
FYI, you might want to look at the thread linked in https://github.com/gjoseph92/stackstac/issues/112. It's reasonable to expect that stackstac's bounds may differ slightly from what you get if you open the GeoTIFFs directly, because we're working with STAC metadata, which could be inaccurate (like in #132). However, this is such a significant mismatch, I'd guess something else is going on here.
I probably won't be able to look into this for a couple weeks. Would you or anyone else (@TomAugspurger @scottyhq?) be interested in taking a look in the meantime?
Gotcha, thanks for the reference @gjoseph92. So stackstac
array bounds and shapes are estimates based on STAC metadata and won't necessarily match file bounds exactly--makes sense. The difference in y
is negligible then, but the huge x
error probably indicates a) incorrect item metadata or b) correct metadata somehow being misinterpreted.
I'll take a look and see if I can track down a problem, but any insights from folks who are more familiar with stackstac
or STAC are definitely welcome.
are estimates based on STAC metadata and won't necessarily match file bounds exactly
I think they should match exactly. But it's possible there's a mismatch. Are you able look at the gdalinfo
of the asset and compare it to the STAC metadata?
Looks like there's a pixel size discrepancy causing the issue: 0.00001
according to STAC vs. 0.000277
according to GDAL.
Manually editing the proj:transform
with the correct pixel size fixes the x
bounds and shape being wildly off. The off-by-one discrepancy in the shape
arises from geom_utils.snapped_bounds
rounding min
coords down and max
coords up, which can be disabled with snap_bounds=False
. Looks like that's as-designed, @gjoseph92?
@TomAugspurger, want me to open an issue on Planetary Computer for the 3dep
STAC transform issue? It seems to affect 10m
items in that collection as well.
Thanks. I opened https://github.com/stactools-packages/threedep/issues/9, which is what we used to generate the STAC metadata. We'll look into it and see what's up.
The off-by-one discrepancy in the
shape
arises fromgeom_utils.snapped_bounds
roundingmin
coords down andmax
coords up, which can be disabled withsnap_bounds=False
. Looks like that's as-designed, @gjoseph92?
Sorry I missed this—yes, that's exactly the point of snapped_bounds
. I feel like snap_bounds=True
is usually what you want, curious if you agree or if that's a bad default.
So sounds like the takeaway is that STAC metadata was off, leading to unexpected results? Nothing to do on the stackstac end?
@aazuspan I'd be curious to confirm—even though the stackstac array was a wildly different shape than you expected, was the data still reasonable? I assume that if you specified resolution= 0.000277
(and maybe bounds too?), you'd get what you wanted?
Thanks for following up! Yeah, STAC metadata plus my confusion about snap_bounds
was the problem, not stackstac
. It looks like the STAC for those items hasn't been updated yet, but manually setting the correct proj:transform
before stacking confirmed that it should work once that's up. I wasn't able to get usable data without setting the proj:transform
. Specifying resolution
and bounds
gave the correct array shape, but the data was ~95% nans. I can post reproducible examples of what I was doing if you're interested.
I do think defaulting to snap_bounds=True
can lead to surprising behavior with the array shape and bounds not matching STAC/GDAL. Is there a downside to leaving fractional offsets? My preference would be to return the data generally "as-is", without snapping by default, unless there was a reason that might cause problems.
Specifying
resolution
andbounds
gave the correct array shape, but the data was ~95% nans
Ah. I wouldn't be surprised if stackstac thought that certain chunks were entirely out of bounds for a certain asset, so it skipped opening and reading them entirely. I bet if you increased the spatial chunksize a ton (like maybe even to -1), you'd get the data you expect, but what's the point of that?
Is there a downside to leaving fractional offsets? My preference would be to return the data generally "as-is"
Especially for users newer to geospatial, not dealing with fractional offsets seemed more approachable and intuitive to me. And in many cases, data "is-is" goes out the window when you're stacking—you usually are warping most assets to a common grid and bounding box anyway, at least for sensors like landsat/sentinel/etc. So wherever there's been a tradeoff between usability and preservation of original data, I've gone for usability, since stackstac is less a tool for retrieving original data than for combining lots of pieces of original data together to get them into a format that's useful to you for downstream analysis.
I bet if you increased the spatial chunksize a ton (like maybe even to -1), you'd get the data you expect
Yep, chunksize=-1
plus specifying bounds
and resolution
did the trick. Good to know there's a workaround for incorrect STAC metadata, without having to manually edit it.
I've gone for usability, since stackstac is less a tool for retrieving original data than for combining lots of pieces of original data together
Totally reasonable. I was thinking in the context of nice, evenly-spaced 3dep
tiles, but snapping definitely makes more sense when you have irregular, overlapping scenes like Sentinel.
Thanks for your help, and for building stackstac
! I'm ready to close this, unless you'd rather keep it open in case someone else runs into it before the STAC is fixed.
Looks like https://github.com/stactools-packages/threedep/issues/9 is closed, but @TomAugspurger are we still waiting for the data to be reprocessed? If that will happen in the next week or so, let's leave it open, otherwise we can close.
Yeah, the items need to be regenerated with the patched stactools and re-ingested. No ETA on when that'll happen right now.
I've opened and pinned https://github.com/gjoseph92/stackstac/issues/152 for these sorts of issues, so hopefully others running into the same thing, or similar things in the future, will find that. So I'm closing this specific one now.
Hi @gjoseph92, I'm using
stackstac.stack
with two adjacent COGs from the Planetary Computer3dep-seamless
collection, and the output bounds and shape are incorrect. If I load them manually withxarray
andconcat
them, the bounds are correct.Reproducing
Issues
The bounds for the two arrays are much different and the
stackstac
version doesn't cover the the searchbbox
:The array shapes (ignoring the
time
andband
dims) are also much different:Checking the
proj:shape
property for the items indicates their shape should be3612 x 3612
, so it looks like thestackstac
version is off by 1 in they
dimension and 100k in thex
dimension.Any thoughts on what's going on here? Thanks!
For reference, I'm running
stackstac=0.4.1
andxarray=2022.3.0
.EDIT: This looked like it might be related to #132, but the
proj:transform
of the items seem to be in the correct order: