A key property of content hashes is that they are deterministic. This allows you
to handle Python objects and automatically arrive at the right UID (hence,
storage location) behind the scenes, without having to think or make decisions
about names or storage. E.g., a simple example would be
@op()
def f(x) -> int:
return x + 1
# on Monday...
with run(storage):
f(23)
# on Tuesday...
with run(storage):
f(23)
Since the hashing is deterministic, this will correctly figure out that f was
already executed with this input.
However a problem can happen in a few ways:
when you have a custom object you're hashing that depends on some library. You
update the library version; something about the internal state of the object
representation changes; you now may end up with a different content hash
you change the version of the tool you use to magically hash (almost) any
Python object (joblib currently).
you have multiple people accessing the same storage that use different library
versions. This leads to different content hashes for equivalent objects.
This can be very bad since you could trigger a completely new computation of a
pipeline.
This issue's goal is to figure out what our constraints for this are and design a solution.
Some very rough possibilities:
record library versions and flat-out refuse to compute if a change is detected
(too much?)
enforce canonically e.g. json-serializable values only, e.g. Python's native
types and recursive combinations thereof + arrays/series/dataframes. Is it
good enough?
let people implement their own content hashes for custom objects alongside
objects that are easy to serialize. This can always be added later.
A key property of content hashes is that they are deterministic. This allows you to handle Python objects and automatically arrive at the right UID (hence, storage location) behind the scenes, without having to think or make decisions about names or storage. E.g., a simple example would be
Since the hashing is deterministic, this will correctly figure out that
f
was already executed with this input.However a problem can happen in a few ways:
joblib
currently).This can be very bad since you could trigger a completely new computation of a pipeline.
This issue's goal is to figure out what our constraints for this are and design a solution. Some very rough possibilities: