Cache Management

[x] allow custom cache validation
- cached results containing/wrapping a failed future shouldn't be kept in cache
[x] failed Downloads should be dropped from cache
[x] when restarting a failed Download, its previously successfully downloaded parts should be kept/reused and only the failed parts/chunks retried
[x] add ttl-based cache clearing
[ ] make cache persistable / serializeable
[x] use factory for StudIP API data model classes based on object UIDs, preventing duplicate objects
[x] directly cache the results of the StudIP API Session getter functions
[x] only persist the now unique instances of the StudIP API data model classes
[x] reuse expired cache-value as fallback when new attempt fails too often or takes to long
[x] implement circuit-breaker algorithm for checking for network connectivity and deciding whether (or how often) a function should be retried or the expired value should be reused

Task Management

[ ] not being able to login in when starting loop will always prevent startup when offline
[ ] session termination during runtime is not handled
[x] store cached values on disk periodically and load then on program start
[ ] allow eager loading of certain directories (with lower priority than user/FUSE operations)
[ ] implement framework for lifecycle tasks
1. startup tasks (e.g. restore model),
2. low priority background tasks (e.g. reset semester selection, save model),
3. required/dependency (cached) tasks (e.g. login) and
4. shutdown tasks (e.g. reset semester selection, save model)

Future Work

[x] cached functions and download parts should be retried immediately on temporary network failures
[x] use ephemeral caching on all other functions (if really needed)
[ ] test fuse functionality, cache logic and API functions
[ ] allow user to switch between online and offline operation
- provide two hidden directories that also contain the full directory tree, but using only cached values (/.cached/*) / only online values (/.online/*)
- or make appindicator allowing to show and switch modes
[ ] make all caching and circuit-breaker related values (timeouts, thresholds,...) configurable
[ ] properly implement and tune HTTP timeout and status code handling

N-Coder commented 6 years ago

Restructured issue description, initial description was

[x] allow custom cache validation

cached results containing/wrapping a failed future shouldn't be kept in cache

[x] failed Downloads should be dropped from cache

[x] when restarting a failed Download, its previously successfully downloaded parts should be kept/reused and only the failed parts/chunks retried

[x] add ttl-based cache clearing

[ ] cached functions should be retried on temporary network failures

[ ] reuse expired cache-value as fallback when new attempt fails too often or takes to long

this requires an algorithm for checking for network connectivity and deciding whether (or how often) a function should be retried or the expired value should be reused

should provide two hidden directories that also contain the full directory tree, but using only cached values (/.cached/*) / only online values (/.online/*)

[ ] allow eager loading of certain directories (with lower priority than user/FUSE operations)

[x] also store cached values on disk and load then on program start

[ ] all the above should allow offline operation

Session management is also not ready for offline operation:

[ ] not being able to login in when starting loop will always prevent startup

[ ] session termination during runtime is not handled

Blindly caching everything mit not be efficient enough, so for being able to persist cached values we need to

[x] use factory for StudIP API data model classes based on object UIDs, preventing duplicate objects

[x] directly cache the results of the StudIP API Session getter functions

[x] only persist the now unqiue instances of the StudIP API data model classes

[ ] use ephemeral caching on all other functions (if really needed)

[ ] this should also allow simple Mocking of the API leading to better testability

N-Coder commented 6 years ago

Awesome List

with generic tools to solve various of our problems here

Model Data / Schema Instance Repository

Model Data / Ephemeral Data Caching

async_lru
aiocache
shared / cached tasks in trio (TODO comment)

HTTP Caching

N-Coder / studip-fuse

Improved caching for offline operation #1

Cache Management

Task Management

Future Work

Awesome List

Model Data / Schema Instance Repository

Model Data / Ephemeral Data Caching

HTTP Caching

(HTTP) Retrying

Downloads

Other Event Loops

Utils