issues
search
aws
/
fmeval
Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
153
stars
40
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
[Feature Request] Added metric to factual knowledge + unit/integration tests
#302
kirupang-code
opened
8 hours ago
0
fix: register placeholder_to_record_key in GeneratePrompt transform
#301
xiaoyi-cheng
closed
1 day ago
1
feat: add validate_prompt_template util
#300
xiaoyi-cheng
closed
2 days ago
0
fix: Context precision score computation
#299
awsvmaringa
closed
3 days ago
1
feat: faithfulness handle no statements edge case
#298
xiaoyi-cheng
closed
1 week ago
0
feat: add error field in EvalScore
#297
xiaoyi-cheng
closed
1 week ago
0
build(deps-dev): bump pdoc from 14.5.0 to 14.5.1
#296
dependabot[bot]
closed
5 days ago
0
feat: add answer relevance algo
#295
xiaoyi-cheng
closed
1 day ago
2
feat: add col_to_validate to evaluate_dataset
#294
oyangz
closed
1 week ago
1
feat: support embedding model runner
#293
xiaoyi-cheng
closed
1 week ago
0
build(deps): bump urllib3 from 1.26.18 to 1.26.19
#292
dependabot[bot]
closed
1 week ago
2
feat: add faithfulness eval algo
#291
xiaoyi-cheng
closed
1 week ago
0
merge main into rageval branch
#290
oyangz
closed
3 weeks ago
1
feat: add context precision for context quality eval algo
#289
oyangz
closed
1 week ago
2
feat: modify GeneratePrompt transform to take placeholder_dict
#288
xiaoyi-cheng
closed
3 weeks ago
0
build(deps): bump tornado from 6.4 to 6.4.1
#287
dependabot[bot]
closed
1 week ago
1
ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead.
#286
aakash086
closed
1 week ago
4
build: bump fmeval version to 1.0.4
#285
danielezhu
closed
1 month ago
0
[Feature] Add callback mechanism to evaluation
#284
acere
opened
1 month ago
1
Support multiple data configs in evaluate
#283
athewsey
closed
2 weeks ago
1
Fix comparisons for string enumerations
#282
athewsey
closed
1 week ago
2
feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs
#281
keerthanvasist
closed
1 month ago
1
[Feature] Add system metrics collected during evaluation to eval_output
#280
acere
opened
1 month ago
4
build(deps): bump requests from 2.31.0 to 2.32.0
#279
dependabot[bot]
closed
1 month ago
4
[Feature] Streaming results/progress and summary metrics for faster feedback
#278
athewsey
opened
1 month ago
1
chore: update BarPlotCell unit test
#277
danielezhu
closed
1 month ago
0
chore: add support for proprietary models
#276
shrestha-bikash
closed
1 month ago
0
build: bump fmeval version to 1.0.3
#275
danielezhu
closed
1 month ago
0
fix: update pinned sagemaker python sdk version and get_user_agent_extra util function
#274
danielezhu
closed
1 month ago
0
feat: allow placeholder dict for prompt composer
#273
xiaoyi-cheng
closed
1 month ago
0
build(deps): bump jinja2 from 3.1.3 to 3.1.4
#272
dependabot[bot]
closed
1 month ago
0
build(deps): bump tqdm from 4.66.2 to 4.66.3
#271
dependabot[bot]
closed
1 month ago
0
build(deps): bump sagemaker from 2.216.1 to 2.218.0
#270
dependabot[bot]
closed
1 month ago
2
[Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency
#269
athewsey
opened
2 months ago
1
[Feature] JSON export/import for EvalOutput classes
#268
athewsey
opened
2 months ago
3
feat: fetch log probability jmespath from JS metadata
#267
keerthanvasist
closed
1 month ago
0
feat: add target_context to dataset columns
#266
oyangz
closed
1 month ago
2
docs: update telemetry-related info in README and docstrings
#265
danielezhu
closed
2 months ago
0
fix: fix patching in unit tests
#264
danielezhu
closed
2 months ago
0
build: bump fmeval version to 1.0.2
#263
danielezhu
closed
2 months ago
1
feat: add fmeval-specific user agent header to botocore config for telemetry purposes
#262
danielezhu
closed
2 months ago
0
docs: add syntax highlighting
#261
connorads
closed
2 months ago
1
docs: create Github Actions workflow for generating docs via pdoc
#260
danielezhu
closed
2 months ago
0
test: update matplotlib version and figure cell init test
#259
oyangz
closed
2 months ago
0
chore: update lib versions based on dependabot recommendation
#258
keerthanvasist
closed
2 months ago
0
build(deps): bump aiohttp from 3.9.3 to 3.9.4
#257
dependabot[bot]
closed
2 months ago
2
chore: simplify botocore/boto3-related util code
#256
danielezhu
closed
2 months ago
0
build: bump fmeval version to 1.0.1
#255
danielezhu
closed
2 months ago
0
fix: update output record key validation logic in validate_call
#254
danielezhu
closed
2 months ago
0
Revert "build: bump fmeval version to 1.0.1"
#253
oyangz
closed
2 months ago
0
Next