I was driving the other day and sort of mulling over the architecture of the system, and realized that having Job totally own its key based on external data is only going to get more annoying when we add recursive dependencies. A simpler architecture would be to let the coordinator coordinate the right key all in one spot.
So where before we would ask Job to calculate the hash for both its direct fields and pass in a Dict of input file hashes, now we ask Job to calculate a key for only the direct fields, and then build on it to get the full hash.
I also went ahead and documented the approach! Should pair well with #46, although these PRs don't depend on each other.
@bhansconnect or @rtfeldman, any chance one of y'all could look at this sometime this week? It's ready to go (and I've just resolved the merge conflict.)
I was driving the other day and sort of mulling over the architecture of the system, and realized that having
Job
totally own its key based on external data is only going to get more annoying when we add recursive dependencies. A simpler architecture would be to let the coordinator coordinate the right key all in one spot.So where before we would ask Job to calculate the hash for both its direct fields and pass in a Dict of input file hashes, now we ask Job to calculate a key for only the direct fields, and then build on it to get the full hash.
I also went ahead and documented the approach! Should pair well with #46, although these PRs don't depend on each other.