Transition Discovery into a stage

[jeremyfowers]

Note to reviewers

A large amount of cleanup tasks are already documented in #168

Summary of Changes

• Discovery is now a stage called discover, which takes one input python script, discovers the models within, and passes one model/inputs to the next Stage.

• New feature: all calls to turnkey have a stats.yaml and state.yaml file now, even if discovery fails.

• New feature: discover is no longer a mandatory way to start the sequence.

  • onnx-load is now an alternative way to start the sequence, which does not invoke the discovery code path whatsoever.
  • In the future we can add more entry points! Load-build-from-cache is a good one. We can also add bespoke entry points for specific model types (e.g., LLM), rather than trying to handle everything generically as a pytorch model or onnx model.

• A lot of functionality has moved out of the turnkeyml.analysis module

  • turnkeyml.analysis is now only responsible for discovering models. It no longer does error handling, stats, nor does it call build_model()
  • The main API now calls build_model() and saves the high-level stats
  • Sequence is responsible for all error handling.

• Only one model is passed from discover to subsequent stages.

  • By default, it is the first model in the input script.
  • Users can specify a model with the ::HASH postfix, as before, however now only one HASH is allowed.
  • This allows us to change the naming scheme for builds:
  • turnkey -i SCRIPT.py discover will name the build SCRIPT
  • turnkey -i SCRIPT.py::HASH discover will name the build SCRIPT_HASH

• The concept of an evaluation_id has been eliminated.

  • stats.yaml is flat now; there is no evaluations section.
  • The stats-related APIs have changed to simply save_stat(), instead of save_model_stat() and save_model_eval_stat()

• The concept of a ModelType enum has been eliminated.

  • Code that cares about the type of the model should just test the type of the model in-situ instead of relying on global model type detection.

[ramkrishna2910]

All cache folder names include _turnkey in the end Is that necessary?

[jeremyfowers]

All cache folder names include _turnkey in the end Is that necessary?

@ramkrishna2910 its not all models. The examples/cli/scripts models all have a label author::turnkey and the author name is auto-populated into the build dir