search

premAI-io / benchmarks

🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
MIT License
117 stars 5 forks source link

Addition of Mistral in HF Transformers with Quality comparison README and Logging Improvements #167

Closed Anindyadeep closed 2 months ago

Anindyadeep commented 2 months ago

This PR Adds some improvements to

  1. Base class, by fixing some bugs, and addition of utilities to make chat templates
  2. Adds Mistral support in benchmarks
  3. Quality comparison readme in engine specific readme
  4. Finally template for mistral containing Mistral specific Performance logs

Please note (Proposal):

We should Merge this PR to a different staging branch. Reason being, README will be broken after this which can also reflect in main branch. And since we changed the model from Llama 2 to LLama 2 Chat, we get bit different results. So proposal is, up next all the PRs, gets merged under staging branch. Once most of all the benchmarks are added, we can merge staging to main.

Update:

There are some improvements introduced in this PR for Logging (under utils.py) Where the main focus was the following:

  1. Small bug fixes where quality.json was taking redundant values
  2. Generates a quality.md README from quality.json. The reason being, it is super hard to copy paste the output in readme manually, so better to copy paste the generated README.
  3. Integrating those in PyTorch Benchmark

This PR also solves issues #162 #157

nsosio commented 2 months ago

@Anindyadeep I've created a dev branch where you can create all the PRs and then once everything is done we can merge dev into main. Can you update the PRs against dev?

Anindyadeep commented 2 months ago

@Anindyadeep I've created a dev branch where you can create all the PRs and then once everything is done we can merge dev into main. Can you update the PRs against dev?

Awesome, I changed this PR to dev branch, lemme know if we can merge this one and all looks good