Download with Cache

[jdunkerley]

Improve support for web requests with an in built cache.

• Should read the caching headers from response.
• Only cached for the session with temporary file cleaned up on exit.
• Only cached for GET requests.

Discuss engine-side caching with @JaroslavTulach

Should be able to be disabled by the end user (at a request or higher level).

• i.e. Data.fetch and equivalent have options to control the caching.

Look to use Channels or another modern Java fetch techniques.

[JaroslavTulach]

• Where should the cached files be stored? In projects data directory?

Look to use Channels or another modern Java fetch techniques.

• Should we download multiple resources in parallel? Then virtual threads would be our choice.

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-02):

Progress: Cache key, secrets handling, HTTP response payload It should be finished by 2024-10-09.

Next Day: Caching logic in EnsoSecretHelper

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-03):

Progress: http request caching: built cache key + lookup logic It should be finished by 2024-10-09.

Next Day: TTL, LRU, other cache things

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-04):

Progress: learning all about TTL, adding download and TTL features to the test web server It should be finished by 2024-10-09.

Next Day: more of that, tests

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-08):

Progress: Calculate TTL based on response headers; implemented TTL and cache logic; got existing tests passing It should be finished by 2024-10-09.

Next Day: cache-specific tests

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-09):

Progress: Writing tests for TTL and cache cleanup It should be finished by 2024-10-10.

Next Day: more of the same

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-10):

Progress: Tests for max-age/Age headers and TTL, mocking time It should be finished by 2024-10-11.

Next Day: secrets in URI and headers

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-11):

Progress: Planning for Data.read_many; HTTP cache tests It should be finished by 2024-10-14.

Next Day: secrets in URI and headers

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-14):

Progress: Cache expiry / cleanup tests It should be finished by 2024-10-15.

Next Day: secrets in URI and headers

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-15):

Progress: Cache expiry; secrets in URI and headers; finished tests It should be finished by 2024-10-16.

Next Day: cleanup, PR

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-16):

Progress: HTTP cache docs, extra tests, PR It should be finished by 2024-10-17.

Next Day: extended round range benchmark regression

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-17):

Progress: only run expiry when adding cache entries; review It should be finished by 2024-10-21.

Next Day: use channels for copying

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-21):

Progress: Separating cache into secrets, http, and storage layers It should be finished by 2024-10-22.

Next Day: use channels for copying

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-22):

Progress: refactored cache into secrets, http, and lru layers; removed apache commons + implemented dowload directly It should be finished by 2024-10-23.

Next Day: benchmark regression

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-23):

Progress: cache review; testing parameters class; check response data in tests It should be finished by 2024-10-24.

Next Day: benchmark regression

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-24):

Progress: cache review; debugging audit + s3 test failures; cache planning; delete / close cache file reorganization; fixed cache policy defaulting logic; fixed Float.parse benchmark regression It should be finished by 2024-10-25.

Next Day: caching or parallel download

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-25):

Progress: caching planning; Float.parse regression PR; SQLite file cleanup; added warning when using Floats as dict keys; regex benchmarks It should be finished by 2024-10-28.

Next Day: finish those

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-28):

Progress: test fix for float parse benchmark regression fix; investigating test failure for float dictionary key warning It should be finished by 2024-10-29.

Next Day: small prs; download caching modes

[enso-bot[bot]]

Greg Travis reports a new STANDUP for today (2024-10-29):

Progress: LRU caching test failures; added no_warning to Dictionary.insert to distable double dictionary key warning for distinct tests; misc review items It should be finished by 2024-10-30.

Next Day: finish those

[enso-bot[bot]]

Greg Travis reports a new STANDUP for the provided date (2024-10-07):

Progress: London trip preparation; learning about cache control headers; implemented max-age in test server, and reading in cache client It should be finished by 2024-10-30.

Next Day: store + use TTL parameter