Smart/adaptive caching - Githubissues

Open-EO / openeo-aggregator

openEO driver that combines multiple other drivers

https://open-eo.github.io/openeo-aggregator/

Apache License 2.0

1 stars 1 forks source link

Smart/adaptive caching #74

Closed soxofaan closed 1 year ago

soxofaan commented 2 years ago

Spin-off from #2

smarter caching, to handle temporary glitches better

if cache is outdated: still keep it if it can not be updated yet due to hard failure

if there is a minor failure in one of the back-ends: cache the result with shorter TTL than default

comment by @m-mohr :

This is basically what we already do in the openEO Hub, but it's all JS with a MongoDB in the background and a daily crawling through a cron job. There we have some logic implemented for such cases where the data is not directly cleared in the db until new data successfully comes in, but clears it after a certain amount of failures...

soxofaan commented 2 years ago

I'm starting to think that it will become quite complex to implement this "smart" logic in the current traditional caching layer approach (first check cache and (re)calculate if not available or outdated). As suggested by @m-mohr it's probably easier to work with a background job that regularly regenerates all necessary metadata and pushes this to central storage, to be used by all workers. The logic for workers is than limited to fetching the metadata, without need for complicated caching rules and other mechanisms.

jdries commented 1 year ago

design somewhat similar to jobtracker

soxofaan commented 1 year ago

This ticket is roughly done:

added "console_script" openeo-aggregator-prime-caches to prime caches, e.g. docker style: docker run --rm -e OPENEO_AGGREGATOR_CONFIG=/home/openeo/aggregator/conf/aggregator.dev.py openeo-aggregator:latest openeo-aggregator-prime-caches
Set up a Nifi workflow to run this every 10 minutes for openeo Platform prod and dev

Still to do as follow up:

error handling, log routing for nifi workflow
the znode version of caches in zookeeper is now bumped every 10 minutes, is there a version limit? Or can we skip writing caches if existing value is still good?
improve test coverage of prime_caches.py

soxofaan commented 1 year ago

I think we can close this now.

Apart from the implementation of the openeo-aggregator-prime-caches tool in openeo-aggregator project, I also had to set it up in nifi for scheduled runs and had to spend some time to get logging working in kibana