Open TomAugspurger opened 1 year ago
The processing information (dates/versions) is also in the native metadata for the observation. This has been a contested subject for awhile. This could be a good solution.
CI is passing now.
If you use the version extension, then you will need the processing timestamp in the ID as you'll need two distinct Items which you can link between. Storing them under the same ID would conflict with the unique ID constraint. I think this should be made more clear in the description and the provided solution with using the version extension and the same IDs doesn't work with the unique ID best practice.
It all depends on the catalog implementation. With a static catalog, you can still use the same id but with a different path including the version. The reference in the collection will be the "latest" version with the unique id. Then in the item, you link to the previous version, still with the same id but at a different path that would include the version. In STAC API, this is even simpler using the version API extension
In my understanding, the main concept is that an item is always unique regardless of it's version.
Yes, but the spec says:
It is important that an Item identifier is unique within a Collection, and that the Collection identifier in turn is unique globally. Then the two can be combined to give a globally unique identifier. Items are strongly recommended to have Collections, and not having one makes it more difficult to be used in the wider STAC ecosystem. If an Item does not have a Collection, then the Item identifier should be unique within its root Catalog or root Collection.
and the best practice adds:
One of the key properties is the ID. [...] they just need to be sure it is globally unique, so may need a prefix.
That's what has been written in the spec and just a paragraph above the addition. Reading this addition then is contradicting and confusing. So this should be better explained and the proposed solution with the version extension should be clarified or the uniqueness constraint needs to be weakened.
What is proposed is not contradicting the principle of uniqueness of the ids. You can manage multiple version of the same STAC Item with a unique id but 2 different files. In a collection or globally, there is still a unique STAC Item. Then it is up to the implementation to manage which version to get according to the link or the API.
On the other hand, this is for sure not what is done de facto within space agencies. Most of them including NASA and ESA with LANDSAT, Sentinels and many other includes the processing id, date or archive version in the filename.
If you use the version extension, then you will need the processing timestamp in the ID as you'll need two distinct Items which you can link between.
Oh, I may have misunderstood the version extension. I thought you had had two items with the same ID: item-a
(version 0) and item-a
(version 1). And then the latest version would be available at /collections/<collection>/item-a
and would include a link to the old version at (e.g.) /versioned/<collection>/item-a/
(I don't know exactly what the path would be).
I guess going back to the thing that originally motivated this: Say you have some software that generates a level-2 product from level-1 data (like sen2cor). If I run that at 8:00 and again at 9:00, the actual data assets should be byte-for-byte identical. And while the filenames might differ because they have a processing time, I'd argue that the STAC ID should not include the processing time.
It's a bit more complicated when talking about changes to the actual processing software rather than just different processing times. In that case the outputs might not be byte-for-byte identical and so you could argue that item-a
(processed with version 1) is distinct from item-a
(processed with version 2). But as a best practice I'd say we probably want people using the latest and so the item ID should not include that information.
As a user, I (probably) want the "latest" (best) version of the assets for a particular spatio-temporal footprint. I (probably) don't want to have to think about choosing between multiple items with the same spatio-temporal footprint. And for the less typical case where you do want the "old" version, we have the version extension.
As a user, I (probably) want the "latest" (best) version of the assets for a particular spatio-temporal footprint. I (probably) don't want to have to think about choosing between multiple items with the same spatio-temporal footprint.
I agree with this as a motivating principle, and think that this could eventually be hardened into a Best Practice, i.e.: "Within a single collection, it is considered best practice to only have one non-deprecated item with a given spatio-temporal footprint." (see what I did there w/ the non-deprecated thing? More on that later)
@m-mohr is correct that the unique-ID constraint forces us to include some sort of version information (whether it's processing datetime, an incrementing integer, a hash, whatever) in the item ID if we want to support item versions within a single collection. Which leads to three possible solutions (as I see it):
I think 1 is fine, but I think we can do better. My proposal:
deprecated=True
cc @matthewhanson, @pjhartzell, @ircwaves, and @arthurelmes (who joined me in a chat about this topic this week)
Thanks Pete, your proposal sounds pretty solid. I think there are some details to work out (does iterating the items in a collection include deprecated items?) but it sounds workable. It solves my main issue with processing information in the IDs today and has the advantage of not silently changing the assets referenced by an item ID (at least I think that's an advantage... I suppose it's not always clear).
Dev call:
Is there an update on the PR?
This proposes a change to the Item ID best practices, based on some experiences and conversations with folks like @gadomski.
In my experience, many upstream data providers (USGS / landsat & MODIS, Copernicus / Sentinel,) include some kind of "processing timestamp" in their IDs. They'll occasionally reprocess assets, leading to new upstream IDs with the same "acquisition" timestamp but a new "processing" timestamp (what happens to the old assets varies, but I think doesn't matter for this discussion).
It's fundamentally ambiguous whether a reprocessed item is the "same" as an existing item. But I think the best recommendation is that the new, reprocessed item / assets should replace the old item / assets. That satisfies the common case of "Give me the item at this datetime over this area". If the processing datetime is included in the item ID then a provider would either
Between the versioning and processing extensions, STAC has all the building blocks to handle this elegantly. So this PR updates the recommendation to use those instead of stuffing a processing timestamp in the item ID.