Content hashing and library versions

[amakelov]

A key property of content hashes is that they are deterministic. This allows you to handle Python objects and automatically arrive at the right UID (hence, storage location) behind the scenes, without having to think or make decisions about names or storage. E.g., a simple example would be

@op()
def f(x) -> int:
    return x + 1

# on Monday...
with run(storage):
    f(23)

# on Tuesday...
with run(storage):
    f(23)

Since the hashing is deterministic, this will correctly figure out that f was already executed with this input.

However a problem can happen in a few ways:

• when you have a custom object you're hashing that depends on some library. You update the library version; something about the internal state of the object representation changes; you now may end up with a different content hash
• you change the version of the tool you use to magically hash (almost) any Python object (joblib currently).
• you have multiple people accessing the same storage that use different library versions. This leads to different content hashes for equivalent objects.

This can be very bad since you could trigger a completely new computation of a pipeline.

This issue's goal is to figure out what our constraints for this are and design a solution. Some very rough possibilities:

• record library versions and flat-out refuse to compute if a change is detected (too much?)
• enforce canonically e.g. json-serializable values only, e.g. Python's native types and recursive combinations thereof + arrays/series/dataframes. Is it good enough?
• let people implement their own content hashes for custom objects alongside objects that are easy to serialize. This can always be added later.
• ???

[amakelov]

Closing this as out of scope for now