search

aws-samples / fm-leaderboarder

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts
Apache License 2.0
18 stars 5 forks source link

Fixing calc invalid fields + invalid $feature token #16

Closed Edo-Friedman closed 2 months ago

Edo-Friedman commented 2 months ago

Issue #, if available: Doesn't work due to various exceptions

Description of changes:

  1. Adding explanation for generating (mandatory) Hugging Face access token in README.md
  2. _models_totest in _summariziationexample.ipynb contained "$feature" token in the "_prompttemplate" field, however, fmeval/eval_algorithms/init.py: _DEFAULT_PROMPT_TEMPLATE = "$modelinput", replaced "$feature" with "$model_input"
  3. Fixing exceptions in _pricingcalculator.py caused by missing fields (pricing of output tokens, or int value multiplied by float value)
  4. Adding jupyterlab to requirements.txt to enable local execution of Jupyter

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.