search

Element84 / earth-search

Earth Search information and issue tracking
https://earth-search.aws.element84.com/v1
27 stars 2 forks source link

confirmation of spatial and temporal consistentcy of offset for Sentinel-2 COGs #24

Open mikoontz-vp opened 8 months ago

mikoontz-vp commented 8 months ago

Hi all,

First, thank you all so much for this great service that you provide in making the Sentinel-2 data more usable.

I'm hoping to get some clarity on how consistent the application of the new offset is across the catalog. I understand that the intention is for all Sentinel-2 scenes processed to COGs after January 25, 2022 using the /v1 API to have the offset applied (https://github.com/cirrus-geo/cirrus-earth-search/issues/26). I also understand that the earthsearch:boa_offset_applied is the COG metadata property that says whether or not the offset has been applied, but that the actual scale/offset are not yet written to the COG metadata but that it is planned. Is that all correct?

But it also seems like there might be some inconsistency in the offset actually being applied, or perhaps the the metadata doesn't always accurately reflect whether the offset was applied even using the /v1 API for scenes after January 25, 2022 (https://github.com/cirrus-geo/cirrus-earth-search/issues/26#issuecomment-1305578996 and https://github.com/cirrus-geo/cirrus-earth-search/issues/21#issuecomment-1516412353). Is that still true? Or has the offset been consistently applied and the metadata updated for all scenes after January 25, 2022?

Just confirming here: is there any way to tell pixel-by-pixel whether the offset needs to be applied (for instance if we are doing a windowed read that happens to span multiple scenes after having built a .vrt), or do we need to check the scenes first, apply the offset if necessary at the scene level, then proceed?

A second thanks for soliciting community input for how to handle this going forward (https://github.com/Element84/earth-search/issues/23)! I'll add a comment to that thread, but a relevant question for this thread is: once a decision is made, it sounds like we'll need to ensure that our tooling that leans on these COGs should be double checked since the datatype, NAflag, need to apply an offset, and/or data range might differ. Is that right too?

Thanks again, and please let me know if I can clarify any of my questions!

gadomski commented 8 months ago

Hello @mikoontz-vp, thanks for the questions. Check out the latest post to our mailing list for the current and future plans for those offsets and the new sentinel-2 collection: https://us13.campaign-archive.com/?u=a7a7fcb1ce46c4d001fc76289&id=cf7934577f. Tl;dr: we're doing a complete reprocessing of the sentinel-2 archive into a new collection, sentinel-2-c1-l2a. That new collection will not have the offsets applied to the data, but the scale and offset will be set in the TIFF metadata, which should be picked up by most modern tooling. The existing collections will be deprecated and eventually removed.

Hope that helps, and let me know if I missed anything. Also, make sure you're subscribed to our mailing list to get the latest updates: https://element84.us13.list-manage.com/subscribe?u=a7a7fcb1ce46c4d001fc76289&id=38266ef009.