aiverify-foundation / moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
https://aiverify-foundation.github.io/moonshot/
Apache License 2.0
180 stars 34 forks source link

Notebook tutorial & docs are not clear enough on moonshot-data & key concepts #378

Open athewsey opened 4 weeks ago

athewsey commented 4 weeks ago

Some members of our extended team had a real struggle getting started with Moonshot via Python recently, because the basic notebook tutorial and the Moonshot docs collectively:

I suggest:

  1. examples/jupyter-notebook/Tutorial 1 - Basic Workflow - Execute a Benchmark.ipynb should add a bit more guidance & clarity on how to set moonshot data location and what connectors are available.
  2. It feels like the docs website is missing a high-level conceptual guide explaining the abstractions Moonshot defines e.g. Connector, Endpoint, Recipe, Cookbook, etc?
  3. To be honest, I find the whole moonshot-data installation process clunky... If it's a simple git clone, what value do we get by hiding that behind a python -m moonshot -i moonshot-data CLI tool? Seems like it would be better to openly tell users what this step is doing and how moonshot references the assets?

I can propose some edits for 1 in a PR, but 2 & 3 feel like bigger changes...