search

Open-EO / openeo-api

The openEO API specification
http://api.openeo.org
Apache License 2.0
91 stars 11 forks source link

Expiry time for web services #465

Closed m-mohr closed 1 year ago

m-mohr commented 1 year ago

What do you think about adding "expiration date" or something similar to openEO spec for services? As in the idea that we could have services automatically deleted after some time, which would be very useful for on-demand-viewer, especially in case of the Jupyter Notebook, where there is no user action that would trigger the deletion.

Originally posted by @Ardweaden in https://github.com/Open-EO/openeo-web-editor/issues/277#issuecomment-1294722139

m-mohr commented 1 year ago

@Ardweaden To be useful, you'd need to set a expiry date by default and make users explicitly choose "infinity" (or so) on request, because otherwise user will not set any expiry usually. They don't even set more basic metadata often (like a title) and as such I'd think that if expiry is optional, it wouldn't be used too often.

Maybe this is more a policy thing. For batch jobs results are deleted on some back-ends after 30 days without being specified in the API. Services have an "enabled" flag, which could be a way to handle it. For example, services could be disabled after 90 days or (maybe better) 30 days after the last request (if you can track that) and then if they are not re-enabled again after e.g. 30 days, you could completely delete them.

Ardweaden commented 1 year ago

if expiry is optional, it wouldn't be used too often.

It would be used often implicitly, whenever they would use on-demand-viewer.

To be useful, you'd need to set a expiry date by default and make users explicitly choose "infinity" (or so) on request

Given it's also connected to the backend policy of deleting stale stuff, it would be IMO be up to the backend to decide what the default is anyway.

m-mohr commented 1 year ago

Ah, you are thinking to set that automatically for "Run now" requests. Can be done in the Web Editor (although there it tries to delete auotmatically), but is it also planned to add a "run now" functionality to the other clients?

If you set a default, you'd also need to inform the user what the default is. Right now we are already creating frustration with deleting old jobs... https://github.com/openEOPlatform/architecture-docs/issues/275

Ardweaden commented 1 year ago

but is it also planned to add a "run now" functionality to the other clients

Yes, I added it to the Python client so it can can be viewed in Jupyter notebooks. However I didn't find a viable solution other than having the user explicitly delete the service.

If you set a default, you'd also need to inform the user what the default is

Could we actually have this in configuration of the service? That way backend could expose the default in /service_types. Regarding what would be used by clients for "run now" (on-demand-viewer), they can inform the users themselves if they feel the need to, e.g. web editor can have a warning or whatever.

m-mohr commented 1 year ago

Yes, this can be part of configuration. Good idea!

Ardweaden commented 1 year ago

However, now there are no "standard" configuration settings across the backends, so each backend could expose the same setting with different names. So we would have to somehow standardise that for expiration date. And probably make it a required configuration setting.

m-mohr commented 1 year ago

We don't do that in the API, this can be standardized for openEO Platform in configuration only.

Also, we can't make it a required configuration setting as it's a breaking change. This would only be possible in API v2.

Ardweaden commented 1 year ago

We don't do that in the API, this can be standardized for openEO Platform in configuration only.

That makes sense, but I'm not sure how to proceed here. Should we then agree on the name of the configuration setting that "run now"/on-demand-viewer will use to set a shorter expiration time? It's then up to the backend to implement it or not - and what name to use if they do implement expiry - but "run now"/on-demand-viewer will ofc not be setting shorter expiration date if not implemented by its "standard".

m-mohr commented 1 year ago

I'll move this to architecture-docs in openEO Platform so that we can come up with something there.

Edit: Sorry, I can't do that. Please open a new issue there so that the involved parties can agree on a common solution for openEO Platform.

m-mohr commented 1 year ago

Either use configuration in /service_types or check PR #471 for a global alternative.