Offline files support

[lhcorralo]

Hi there!

I am working in a project where some items (and their assets) may not be available inmediatly. For example, if the assets are stored in tapes, they won't be accesible until they are copied to an S3 or similar (ok, not the best example, but I hope the idea is clear).

At the moment we are thinking in two alternatives:

• A property for every item indicating the state of the assets (online/offline, for example). This would require to support PUT requests for a specific item, as the item will be updated.
• Async capabilities: the STAC server would return 202 Accepted if there are offline files, and the cliente would periodically poll the server. When everything is ready, 200 OK would be returned.

I could not find anything in the specification regarding this. I found that similar APIs (finder.creodias.eu, which I think it uses jjrom's Resto as server) use the first approach (despite Resto not accepting PUT requests for an existing item). However, I would like to know if there is an "official approach" for this.

Thank you!

Luis

[jjrom]

Hi Luis - we have this requirement in a bunch of projects and we use a mix of both alternatives.

The « offline » parameter is used as an explicit indicator to the user so he knows which products are directly available and which products required some processing time to be retrieved.

When an offline product is requested an HTTP 202 request is sent back to the client. The poll strategy is to call back the server after 2 seconds, then 4 seconds, then 8 seconds, etc. until an HTTP 200 is sent back and the product downloaded.

The item is then updated server side to set the availability parameter to « online ».

I confirm that finder.creodias.eu uses resto but not a recent version. The current version of resto supports the update of items (using PUT request) ;)

[emmanuelmathot]

Hello @lhcorralo There is a dedicated extension for your use case: https://github.com/stac-extensions/order It allows to specify metadata about data availability through "ordering". The extension describes 2 use cases and the first one seems to be your case. Then it is up to you on how to implement it. I would be glad to have a feedback on that extension as well.

[lhcorralo]

Hi @jjrom

Thank you for your reply! I like the strategy you are using, so I guess I will follow it too :) I wonder if it could be nice to add this somewhere in the documentation. I mean, I think it does not exactly fits into the specification at the moment, but it is a good "best practices" case.

About Resto accepting PUT method, I did a quick test and I could not get it but, well, it was a really quick test, and maybe in an out to date Resto server, so I bet I missed something... I will try again :)

[lhcorralo]

Hi @emmanuelmathot!

I did not know the extension, but it looks promising! I have to read it carefully (I am having some questions, but likely caused by a lack of coffee), but sure, I would be happy of contributing with feedback :)