issues
search
aws
/
fmeval
Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
151
stars
40
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
build(deps): bump idna from 3.6 to 3.7
#250
dependabot[bot]
closed
2 months ago
2
build(deps): bump transformers from 4.37.2 to 4.38.0
#249
dependabot[bot]
closed
2 months ago
2
docs: update README with details on contributing new eval algos
#248
danielezhu
closed
2 months ago
0
[Feature] Image fields for multi-modal models
#247
athewsey
opened
2 months ago
1
[Feature] Multi-variable prompt templates
#246
athewsey
opened
2 months ago
2
fix: fix s3 uri validation for built-in datasets
#245
oyangz
closed
2 months ago
1
build(deps): bump pillow from 10.2.0 to 10.3.0
#244
dependabot[bot]
closed
2 months ago
2
build: bump fmeval version to 1.0.0
#243
danielezhu
closed
3 months ago
0
fix: set default region for boto3 client to access built-in datasets
#242
oyangz
closed
3 months ago
0
feat: update implementation of Toxicity to use Transform-based approach
#241
danielezhu
closed
3 months ago
0
feat: update implementation of PromptStereotyping to use Transform-based approach
#240
danielezhu
closed
3 months ago
0
Updating third party attributions
#239
malhotra18
closed
3 months ago
0
feat: update implementation of FactualKnowledge to use Transform-based approach
#238
danielezhu
closed
3 months ago
0
feat: update implementation of ClassificationAccuracySemanticRobustness to use Transform-based approach
#237
danielezhu
closed
3 months ago
0
feat: update implementation of ClassificationAccuracy to use Transform-based approach
#236
danielezhu
closed
3 months ago
0
feat: update implementation of QAAccuracySemanticRobustness to use Transform-based approach
#235
danielezhu
closed
3 months ago
0
feat: update implementation of QAAccuracy to use Transform-based approach
#234
danielezhu
closed
3 months ago
0
feat: update implementation of SummarizationAccuracySemanticRobustness to use Transform-based approach
#233
danielezhu
closed
3 months ago
0
refactor: update evaluate_dataset to take in a dataset instead of dataset config
#232
danielezhu
closed
3 months ago
0
chore: restore evaluate_sample and evaluate signatures in EvalAlgorithmInterface
#231
danielezhu
closed
3 months ago
0
feat: update implementation of SummarizationAccuracySemanticRobustness to use transform-based approach
#230
danielezhu
closed
3 months ago
0
fix: restore semantic perturbation constants to their original values
#229
danielezhu
closed
3 months ago
0
fix: update GetModelResponse transform to work with any ModelRunner
#228
danielezhu
closed
3 months ago
0
Toxicity transforms
#227
franluca
closed
3 months ago
1
Cannot import FactualKnowledge module
#226
dferguson992
closed
3 months ago
4
feat: updated docstrings
#225
polaschwoebel
closed
3 months ago
0
refactor: move repeated code in evaluate method into util functions and simplify the EvalAlgorithmInterface method signatures
#224
danielezhu
closed
3 months ago
0
feat: example notebook for comparative plotting
#223
polaschwoebel
closed
3 months ago
1
feat: update implementation of GeneralSemanticRobustness to use Transform-based approach
#222
danielezhu
closed
3 months ago
1
build(deps-dev): bump black from 23.7.0 to 24.3.0
#221
dependabot[bot]
closed
2 months ago
2
feat: update GetModelResponse transform to support multiple model invocations on the same input
#220
danielezhu
closed
3 months ago
3
chore: change PromptComposer.PLACEHOLDER from "feature" to "model_input"
#219
danielezhu
closed
3 months ago
1
feat: update various transforms to accept multiple input keys
#218
danielezhu
closed
3 months ago
0
feat: add prompt template to report
#217
oyangz
closed
3 months ago
0
refactor: update Transform API
#216
danielezhu
closed
3 months ago
0
feat: implement transforms for semantic perturbations
#215
danielezhu
closed
3 months ago
0
feat: update implementation of SummarizationAccuracy to use Transform-based approach
#214
danielezhu
closed
3 months ago
0
docs: update README to include information about Windows support
#213
danielezhu
closed
3 months ago
0
fix: update the default prompt templates for the built-in datasets
#212
jmikko
closed
3 months ago
0
feat: implement transforms for summarization accuracy metrics
#211
danielezhu
closed
3 months ago
0
feat: implement helper models used by evaluation algorithms
#210
danielezhu
closed
3 months ago
0
feat: implement Transform and TransformPipeline classes for modular redesign
#209
danielezhu
closed
3 months ago
0
fix: update terminology in README and source code
#208
danielezhu
closed
3 months ago
0
feat: implement validate_call decorator
#207
danielezhu
closed
3 months ago
0
feat: implement Transform and TransformPipeline classes
#206
danielezhu
closed
3 months ago
0
Add support for system prompt and messages API through ModelRunner.predict()
#205
gilinachum
closed
3 months ago
3
feat: implement TransformPipeline
#204
danielezhu
closed
3 months ago
0
fix: add data for example notebook
#203
polaschwoebel
closed
4 months ago
0
Chinese model and content support
#202
weinick
closed
3 months ago
1
feat: implement GeneratePrompt and GetModelResponse utility transforms
#201
danielezhu
closed
4 months ago
0
Previous
Next