search

bacalhau-project / bacalhau

Compute over Data framework for public, transparent, and optionally verifiable computation
https://docs.bacalhau.org
Apache License 2.0
648 stars 85 forks source link

Content addressible functions (jobs) - caching is cool #247

Open lukemarsden opened 2 years ago

lukemarsden commented 2 years ago

ETA: 2023Q1

If we start making the job id the hash of the job spec, we'll get caching for free (although we'll need to test that getting cached results works well with the CLI flow, and we might want a way to request a retry, maybe by bumping some field in the job spec), and the system will also feel more IPFS-y.

Bubblyworld commented 2 years ago

@lukemarsden makes sense to me, but for non-deterministic jobs we probably don't want to cache, right? Since different executions might have different results.

Or I guess to rephrase: where does the caching happen?

lukemarsden commented 2 years ago

hmm yeah, we don't know a priori whether a job is deterministic or not - maybe for jobs where determinism isn't enforced by the runtime, we want to give the user the option to reuse a previous result, or re-run it by bumping a salt in the job spec or something, a bit like how go test will reuse previous results but you can set -count=1 (not obvious UX!) to get it to explicitly rerun

Bubblyworld commented 2 years ago

Mmm, maybe caching should be opt-in for stuff that isn't explicitly deterministic? I can see this being a source of pain for users, the go test -count=1 thing has tripped me up a number of times in my life (run a randomised test a few times, think it's okay, actually it was just caching the first result).

lukemarsden commented 2 years ago

Yeah sure

On Mon, 27 Jun 2022, 10:11 Guy Paterson-Jones, @.***> wrote:

Mmm, maybe caching should be opt-in for stuff that isn't explicitly deterministic? I can see this being a source of pain for users, the go test -count=1 thing has tripped me up a number of times in my life (run a randomised test a few times, think it's okay, actually it was just caching the first result).

— Reply to this email directly, view it on GitHub https://github.com/filecoin-project/bacalhau/issues/247#issuecomment-1167092103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACATUQD6ODUIN74I6ZQWHLVRFV5VANCNFSM5Y2RZWKA . You are receiving this because you were mentioned.Message ID: @.***>

aronchick commented 1 year ago

cc @frrst