use cache on fetches from S3

[jgupta]

It would be good to have a cache for S3 in Moodle.

1. Save expensive network & S3 operations on small files.
2. Save expensive network & S3 operations for high traffic files.

Moodle fetches files from S3 in various scenarios like when pre-signed URLs is not being used or when some files are whitelisted (for scorm?), these fetches from S3 should be cached for increased performance.

[brendanheywood]

What exactly do you mean by 'cached'? Do you mean the file should be copied from s3 back to shared disc? Or local disc?

[jgupta]

Copied to shared disk, maybe to moodledata/cache?

[brendanheywood]

The more you go down this path the more you will end up reinventing a clustered file system itself. If it is shared disc then there is an assumption that shared disc is significantly faster than s3. If you use local disc and you have 10 front ends then you end up with 10x the storage as everything is replicated, and a request goes to a new front end without that file so ends up being slower anyway. #306 might be part of the problem and isthe same as the first option.

This feels like the same root cause as the other issue, the io between the front ends and s3 should ideally be cheap and fast.

[jgupta]

moodledata/localcache is on local storage but moodledata/cache is shared storage. I meant having a cache is shared storage. I agree #306 might fix things for us and this may not be required.

[brendanheywood]

Ok cool thanks, I'll close this one and we'll focus on the others