aws fmeval issues - Githubissues

aws / fmeval

Foundation Model Evaluations Library

http://aws.github.io/fmeval

Apache License 2.0

214 stars 46 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

build: bump fmeval version to 1.0.4

#285 danielezhu closed 6 months ago
0
[Feature] Add callback mechanism to evaluation

#284 acere opened 6 months ago
1
Support multiple data configs in evaluate

#283 athewsey closed 5 months ago
1
Fix comparisons for string enumerations

#282 athewsey closed 5 months ago
2
feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs

#281 keerthanvasist closed 6 months ago
1
[Feature] Add system metrics collected during evaluation to eval_output

#280 acere opened 6 months ago
4
build(deps): bump requests from 2.31.0 to 2.32.0

#279 dependabot[bot] closed 6 months ago
4
[Feature] Streaming results/progress and summary metrics for faster feedback

#278 athewsey opened 6 months ago
1
chore: update BarPlotCell unit test

#277 danielezhu closed 6 months ago
0
chore: add support for proprietary models

#276 shrestha-bikash closed 6 months ago
0
build: bump fmeval version to 1.0.3

#275 danielezhu closed 6 months ago
0
fix: update pinned sagemaker python sdk version and get_user_agent_extra util function

#274 danielezhu closed 6 months ago
0
feat: allow placeholder dict for prompt composer

#273 xiaoyi-cheng closed 6 months ago
0
build(deps): bump jinja2 from 3.1.3 to 3.1.4

#272 dependabot[bot] closed 6 months ago
0
build(deps): bump tqdm from 4.66.2 to 4.66.3

#271 dependabot[bot] closed 6 months ago
0
build(deps): bump sagemaker from 2.216.1 to 2.218.0

#270 dependabot[bot] closed 6 months ago
2
[Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency

#269 athewsey opened 7 months ago
1
[Feature] JSON export/import for EvalOutput classes

#268 athewsey opened 7 months ago
3
feat: fetch log probability jmespath from JS metadata

#267 keerthanvasist closed 6 months ago
0
feat: add target_context to dataset columns

#266 oyangz closed 6 months ago
2
docs: update telemetry-related info in README and docstrings

#265 danielezhu closed 7 months ago
0
fix: fix patching in unit tests

#264 danielezhu closed 7 months ago
0
build: bump fmeval version to 1.0.2

#263 danielezhu closed 7 months ago
1
feat: add fmeval-specific user agent header to botocore config for telemetry purposes

#262 danielezhu closed 7 months ago
0
docs: add syntax highlighting

#261 connorads closed 7 months ago
1
docs: create Github Actions workflow for generating docs via pdoc

#260 danielezhu closed 7 months ago
0
test: update matplotlib version and figure cell init test

#259 oyangz closed 7 months ago
0
chore: update lib versions based on dependabot recommendation

#258 keerthanvasist closed 7 months ago
0
build(deps): bump aiohttp from 3.9.3 to 3.9.4

#257 dependabot[bot] closed 7 months ago
2
chore: simplify botocore/boto3-related util code

#256 danielezhu closed 7 months ago
0
build: bump fmeval version to 1.0.1

#255 danielezhu closed 7 months ago
0
fix: update output record key validation logic in validate_call

#254 danielezhu closed 7 months ago
0
Revert "build: bump fmeval version to 1.0.1"

#253 oyangz closed 7 months ago
0
fix: fix logic in evaluate_dataset to handle BYO inference outputs use case

#252 danielezhu closed 7 months ago
0
build: bump fmeval version to 1.0.1

#251 oyangz closed 7 months ago
0
build(deps): bump idna from 3.6 to 3.7

#250 dependabot[bot] closed 7 months ago
2
build(deps): bump transformers from 4.37.2 to 4.38.0

#249 dependabot[bot] closed 7 months ago
2
docs: update README with details on contributing new eval algos

#248 danielezhu closed 7 months ago
0
[Feature] Image fields for multi-modal models

#247 athewsey opened 7 months ago
1
[Feature] Multi-variable prompt templates

#246 athewsey opened 7 months ago
2
fix: fix s3 uri validation for built-in datasets

#245 oyangz closed 8 months ago
1
build(deps): bump pillow from 10.2.0 to 10.3.0

#244 dependabot[bot] closed 7 months ago
2
build: bump fmeval version to 1.0.0

#243 danielezhu closed 8 months ago
0
fix: set default region for boto3 client to access built-in datasets

#242 oyangz closed 8 months ago
0
feat: update implementation of Toxicity to use Transform-based approach

#241 danielezhu closed 8 months ago
0
feat: update implementation of PromptStereotyping to use Transform-based approach

#240 danielezhu closed 8 months ago
0
Updating third party attributions

#239 malhotra18 closed 8 months ago
0
feat: update implementation of FactualKnowledge to use Transform-based approach

#238 danielezhu closed 8 months ago
0
feat: update implementation of ClassificationAccuracySemanticRobustness to use Transform-based approach

#237 danielezhu closed 8 months ago
0
feat: update implementation of ClassificationAccuracy to use Transform-based approach

#236 danielezhu closed 8 months ago
0

Previous Next