Need documentation / examples using artifact names

[aittalam]

The model artifact from a tuning is run_name-model, which means that if you want to load a model for an eval (e.g. lm-harness) where the run will be updated with eval results, you have to write the following:

model:
  load_from:
    name: "Mistral-7B-Instruct-v0.2-20240212-model" ### NOTE the "-model" suffix here
    project: "davide-testing-flamingo"
    entity: "mozilla-ai"
...
# Tracking info for where to log the run results
tracking:
  name: "Mistral-7B-Instruct-v0.2-20240212"
  project: "davide-testing-flamingo"
  entity: "mozilla-ai"

I think this is ok, but we might want to document it or put it into our examples/tutorials.

Also, all artifacts have a fixed name which is f"{run_name}-{artifact_type}". I currently do not see the need to allow for custom artifact names, but wanted to check your opinion on this.

[sfriedowitz]

A couple comments/questions: 1) Do we want to add the -model tag at the end of the artifacts? We could change that if you feel its not worthfile to have. My thought was that a given run might produce multiple artifacts of different types, so differentiating by name somehow would be useful. 2) Generally, I would imagine when populating your configuration file that you would go to W&B and copy the artifact path from their UI. The WandbArtifactConfig.from_wandb_path constructor might be useful here.

[aittalam]

1. Yes. While in the UI you have artifacts properly split depending on their task (which is great btw! 🙏 ), I think the "full name" might be overlapping if you don't disambiguate them. Example:

   • the full name for https://wandb.ai/mozilla-ai/davide-testing-flamingo/artifacts/evaluation/Mistral-7B-Instruct-v0.2-20240212-evaluation/v0 is mozilla-ai/davide-testing-flamingo/Mistral-7B-Instruct-v0.2-20240212-evaluation:v0
   • the full name for https://wandb.ai/mozilla-ai/davide-testing-flamingo/artifacts/model/Mistral-7B-Instruct-v0.2-20240212-model/v0 is mozilla-ai/davide-testing-flamingo/Mistral-7B-Instruct-v0.2-20240212-model:v0 ... so if we did not have the suffix they'd be the same!

2. Yes, that's a very good idea. At some point we might even have some tooling to help ppl populate their eval config from their run (just provide fine-tuning config, get everything you need except tasks list, build output yaml file)

[sfriedowitz]

build output yaml file

You may be interested in following the entirely in-memory approach to job submission. Take a look at the method BaseLMBuddyConfig.to_tempfile, and the example in examples/notebooks/dev_workflow.ipynb.

It is possible to construct your job config entirely in memory, and only write to a temporary working directory for job submission, meaning you don't need to manually constuct your YAML file.

If we follow this pathway, then we can begin to make use of more utility methods for populating your config classes within the driver script.