Scope out profiling - Githubissues

ManifoldRG / MultiNet

MIT License

7 stars 1 forks source link

Scope out profiling #62

Closed pranavguru closed 3 weeks ago

pranavguru commented 1 month ago

Come up with a comprehensive plan with justifications about how to go about the entire profiling process. Examples of important questions to answer:

Do models on the final list for profiling have fine-tuning APIs? How accessible are the models for the evaluation we plan to conduct?
Estimation of costs and resources required for inference during evaluation

devjwsong commented 1 month ago

Currently, we can only estimate the cost regarding the input tokens since how many tokens would be for the outputs. Both OpenAI and Anthropic costs much higher for the output tokens.

pranavguru commented 3 weeks ago

Closing because more granular tasks listed explicitly (#71 , #70 )