search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

Don't remove batch job metadata #245

Open soxofaan opened 1 year ago

soxofaan commented 1 year ago

https://github.com/Open-EO/openeo-geopyspark-driver/blob/33c996995534605363b3a7cc3d8f1f084ad3446f/openeogeotrellis/cleaner.py#L19-L24

from https://github.com/openEOPlatform/architecture-docs/issues/275:

I'm fine with deleting results after some time as I can recompute them in the worst case, but deleting the whole job (i.e. with the process graph) seems overly destructive. Spending the whole morning just reconstructing an old process graph that I stored in the cloud (and people are trained that stuff in the cloud stays there).

soxofaan commented 1 year ago

We are moving from Zookeeper to ElasticSearch for storage of batch job metadata, which should scale a lot better and eliminate the practical need to clean up old jobs. #246