The key change here is that instead of logging artifact lineages as part of the job entrypoints, I am returning the raw W&B artifacts from the jobs, and logging them at the end of the runs in one location. This offers a few benefits:
1) We reduce a ton of boilerplate code duplication around when/where to initialize the W&B run.
2) Because we are not logging artifacts within the job internals, we no longer need to inject ArtifactLoader interface into the entrypoints. This simplifies the code a good amount.
3) It further standardizes the input/output flow from the entrypoint functions, making them easier for developers to understand and write.
4) It begins to standardize the role of the LMBuddy class. In a follow-up PR, I plan to refactor the organization of the code a bit to add further functionality to this central class. Much of this is motivated by the design of libraries like Ludwig, and the way in which they use their LudwigModel central class.
What's Changing
The key change here is that instead of logging artifact lineages as part of the job entrypoints, I am returning the raw W&B artifacts from the jobs, and logging them at the end of the runs in one location. This offers a few benefits: 1) We reduce a ton of boilerplate code duplication around when/where to initialize the W&B run. 2) Because we are not logging artifacts within the job internals, we no longer need to inject
ArtifactLoader
interface into the entrypoints. This simplifies the code a good amount. 3) It further standardizes the input/output flow from the entrypoint functions, making them easier for developers to understand and write. 4) It begins to standardize the role of theLMBuddy
class. In a follow-up PR, I plan to refactor the organization of the code a bit to add further functionality to this central class. Much of this is motivated by the design of libraries like Ludwig, and the way in which they use theirLudwigModel
central class.