issues
search
aws
/
fmeval
Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
153
stars
40
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Chinese model and content support
#202
weinick
closed
4 months ago
1
feat: implement GeneratePrompt and GetModelResponse utility transforms
#201
danielezhu
closed
4 months ago
0
feat: implement Transform abstract base class
#200
danielezhu
closed
4 months ago
0
Evaluation Algorithm for Recommendations
#199
mccartni-aws
closed
4 months ago
4
docs: add troubleshooting item for OOM errors
#198
keerthanvasist
closed
4 months ago
0
out of memory for summarization accuracy
#197
gilinachum
closed
4 months ago
3
fix: readme
#196
polaschwoebel
closed
4 months ago
0
build: bump fmeval version to 0.4.0
#195
xiaoyi-cheng
closed
4 months ago
0
build(deps): bump aiohttp to fix vulnerability
#194
xiaoyi-cheng
closed
4 months ago
0
fix: add bert_score_dissimilarity description
#193
oyangz
closed
4 months ago
0
chore: remove XSUM dataset from example notebook and integration tests
#192
danielezhu
closed
4 months ago
1
chore: remove xsum dataset and update gigaword description
#191
xiaoyi-cheng
closed
4 months ago
0
chore: pin boto3 and botocore versions
#190
danielezhu
closed
4 months ago
1
chore: Update Ray to 2.9.1
#189
danielezhu
closed
4 months ago
0
fix: Fix example notebook unit tests
#188
danielezhu
closed
4 months ago
1
[fix] Fix example notebook unit tests
#187
danielezhu
closed
4 months ago
0
String enums not comparing as expected
#186
athewsey
closed
1 week ago
3
fix: Toxicity evaluate_sample error message
#185
xiaoyi-cheng
closed
4 months ago
0
feat: add support for non-deterministic models in GeneralSemanticRobustness and add BERTScore Dissimilarity
#184
bilalaws
closed
4 months ago
0
Unable to run in Docker container (unable to register worker with raylet)
#183
athewsey
closed
4 months ago
4
fix: change prompt_template with summarize instruction
#182
taturabe
closed
4 months ago
5
chore: Update readme with installation tips
#181
danielezhu
closed
4 months ago
0
fix: load detoxify model from state dict and upgrade transformers version
#180
oyangz
closed
4 months ago
0
build(deps): bump aiohttp from 3.9.1 to 3.9.2
#179
dependabot[bot]
closed
4 months ago
1
fix: split text by any newline and spaces
#178
franluca
closed
5 months ago
1
fix: update scores description
#177
xiaoyi-cheng
closed
5 months ago
0
fix: unblock release pipeline
#176
xiaoyi-cheng
closed
5 months ago
0
build(deps-dev): bump pillow from 10.1.0 to 10.2.0
#175
dependabot[bot]
closed
5 months ago
1
chore: update devtool all to install first, lint after
#174
keerthanvasist
closed
5 months ago
0
fix: update Ray to version 2.9.0
#173
danielezhu
closed
5 months ago
0
Strip text when computing precision and recall.
#172
bilalaws
closed
5 months ago
0
fix: create single source of truth for dataset column names
#171
danielezhu
closed
5 months ago
0
Dependency issues
#170
aaronkl
closed
5 months ago
1
build(deps-dev): bump jinja2 from 3.1.2 to 3.1.3
#169
dependabot[bot]
closed
5 months ago
1
feat: stringify dataset column contents during data loading
#168
danielezhu
closed
5 months ago
3
feat: conversion of targets to strings
#167
franluca
closed
5 months ago
2
Update f1 score in QA accuracy eval
#166
bilalaws
closed
5 months ago
0
[Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")`
#165
athewsey
opened
5 months ago
3
fmeval is incompatible to properly install on (new) SageMaker Studio Python kernel
#164
athewsey
closed
4 months ago
2
[Feature] LLM-based (QA Accuracy) eval algorithm
#163
athewsey
opened
5 months ago
2
Failed running ./devtool: line 14: pre-commit: command not found
#162
gilinachum
closed
5 months ago
3
fix: replace add_column with map in _generate_prompt_column
#161
danielezhu
closed
6 months ago
0
build: bump fmeval version to 0.3.0
#160
xiaoyi-cheng
closed
6 months ago
0
feat: make sm/br runners easier to subclass
#159
franluca
closed
6 months ago
1
chore: update example notebooks to pip install the fmeval package
#158
danielezhu
closed
6 months ago
1
feat: added the precision and recall metrics for QA accuracy
#157
bilalaws
closed
5 months ago
0
feat: support inference component
#156
xiaoyi-cheng
closed
6 months ago
0
fix: remove s3fs from data-loading code and use boto3 instead
#155
danielezhu
closed
6 months ago
0
fix(security): update to ray 2.8.1
#154
keerthanvasist
closed
5 months ago
0
fix: add proper capitalization for 'SageMaker'
#153
oyangz
closed
6 months ago
0
Previous
Next