[Core feature] Remote cache for local execution in ephemeral environment

[mickjermsurawong-stripe]

Motivation: Why do you think this is important?

• Caching is essential feature of Flyte execution
• Running Flyte local execution on notebook works well in with durable file system, as content is written to this fixed local path
• However, when running so on notebook with ephemeral file system, the written content goes away across different notebook sessions. Local caching then is no-op is this environment.

Goal: What should the final outcome look like, ideally?

Running Flyte local execution in ephemeral filesystem should support caching. Essentially, this local execution in ephemeral environment is similar to the stateless remote execution on K8S where caching relies on external storage. We want to similarly enable remote cache for this ephemeral local execution.

Describe alternatives you've considered

Configuring the ephemeral environment to have persistent storage across notebooking sessions is possible. But for every such environment, we have to implement a bespoke solution.

Propose: Link/Inline OR Additional context

Plumbing the info One problem here is defining the scope of cache key. In remote execution, we have domain/project. For such local execution, client should be able to provide its own ID that determines scope of caching. It's already the intention to make cache path configurable. We can further add the scope of cache key here as well. So the config may look like:

local_cache_path: <local FS path>

or

external_local_cache_path: <s3://...>
local_cache_scope: <...>

Ideally we can also dynamically set local cache scope via env var for convenience. LOCAL_CACHE_SCOPE

Cache implementation

• Option 1/ use the local cache scope as part of the cache key and do lookup on S3
• Option 2/ rely on the existing diskcache impl that excludes cache scope, but upload the entire db artifacts to S3 with the cache scope key...

  s3://my-external-local-cache/<cache_scope>/cache.db
  s3://my-external-local-cache/<cache_scope>/cache.db-shm
  s3://my-external-local-cache/<cache_scope>/cache.db-wal

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[dosubot[bot]]

🤖

It's great to see your interest in improving the caching feature for Flyte local execution in ephemeral environments. Your proposed solution and considerations for defining the scope of the cache key and the implementation options are quite thorough and thoughtful. Your contribution would definitely add value to the platform.

I encourage you to go ahead and create a pull request with your proposed changes. The maintainers will review it and provide feedback. Thank you for your contribution to flyte!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eapolinario]

@mickjermsurawong-stripe , the idea of introducing a local scope is very reasonable, but IMO we should not combine this with interactions with the blob store (e.g. we have to support the blob stores of the big 3 cloud providers).

Would it be ok if we just introduced the local scope, @mickjermsurawong-stripe?

[mickjermsurawong-stripe]

hi @eapolinario! thanks for looking at the issue here! Our problem at hand is that we run local mode in ephemeral environment, and the external cahe -- via blob store -- for local-mode execution is important to us. Could blob store storage here be incrementally implemented first with S3 and then GPC/Azure? If not, i'm also happy to help contribute to all implementations. Do you have concerns on this scope of implementation, or regarding the feature itself of remote cache for local execution?

[pingsutw]

@mickjermsurawong-stripe I think it's pretty reasonable to add in the flytekit. Feel free to open a PR