aws fmeval issues - Githubissues

aws / fmeval

Foundation Model Evaluations Library

http://aws.github.io/fmeval

Apache License 2.0

214 stars 46 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

build(deps): bump tornado from 6.4.1 to 6.4.2

#335 dependabot[bot] opened 1 week ago
0
build(deps): bump aiohttp from 3.10.2 to 3.10.11

#334 dependabot[bot] opened 1 week ago
0
Updated dependencies to improve compatibility with Python3.11

#333 acere opened 2 weeks ago
0
[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation

#332 achad4 opened 3 weeks ago
0
[Feature] Lambda model runner

#331 achad4 opened 3 weeks ago
0
Option to disable BERTScore in QAAccuracy

#330 athewsey opened 1 month ago
0
build: bump fmeval version to 1.2.1

#329 shrestha-bikash closed 1 month ago
0
doc: update README.md to include set up challenges from virtual env

#328 kirupang-code opened 3 months ago
0
build(deps) bump nltk from 3.8.1 to 3.9.1

#327 danielezhu closed 3 months ago
0
build: bump fmeval version to 1.2.0

#326 kirupang-code closed 3 months ago
0
doc: delete ROUGE/METEOR score from QAAccuracy documentation

#325 kirupang-code closed 3 months ago
0
build(deps): bump nltk from 3.8.1 to 3.9

#324 dependabot[bot] closed 3 months ago
2
build(deps): pinned nltk version to address build failure

#323 kirupang-code closed 3 months ago
0
Python 3.11 and 3.12 compatibility?

#322 OlivierBinette opened 3 months ago
3
build(deps): bump aiohttp from 3.9.5 to 3.10.2

#321 dependabot[bot] closed 3 months ago
1
[Fix] Updated notebook rendering as suggested in issue 316

#320 polaschwoebel closed 3 months ago
3
fix: radar plot generation

#319 xiaoyi-cheng closed 3 months ago
1
chore: update urllib3 version

#318 kirupang-code closed 3 months ago
0
refactor: `SummarizationAccuracyMetrics` transform to handle multiple target outputs more efficiently

#317 kirupang-code closed 3 months ago
0
[Bug] Radar plot using plotly not displaying

#316 jcassiojr closed 3 months ago
0
feat: add `BERT_SCORE` to `QAAccuracySemanticRobustness`

#315 kirupang-code closed 3 months ago
0
feat: Add `BERT_SCORE` to `QAAccuracy` and update unit/integration tests

#314 kirupang-code closed 3 months ago
1
build: bump fmeval version to 1.1.0

#313 danielezhu closed 4 months ago
0
chore: rename factual knowledge scores

#312 danielezhu closed 4 months ago
0
feat: update s3 data source for us-isof partition

#311 oyangz closed 4 months ago
1
fix: rename target/retrieved context to `context`, make it a list

#310 xiaoyi-cheng closed 4 months ago
0
fix: update how default payloads get extracted from model spec

#309 danielezhu closed 4 months ago
0
fix: update how default payloads get extracted from model spec

#308 danielezhu closed 4 months ago
0
feat: add configurable param logical_operator (OR/AND) to factual knowledge

#307 kirupang-code closed 4 months ago
0
WIP: source code changes to fix default payloads issue.

#306 danielezhu closed 4 months ago
0
feat: update context to take lists and rename context field

#305 oyangz closed 4 months ago
1
build(deps): bump zipp from 3.19.0 to 3.19.1

#304 dependabot[bot] closed 3 months ago
4
build(deps): bump certifi from 2024.2.2 to 2024.7.4

#303 dependabot[bot] closed 4 months ago
0
feat: add `quasi_exact_inclusion` metric to factual knowledge; change `factual_knowledge` score name to `exact_inclusion`

#302 kirupang-code closed 4 months ago
1
fix: register placeholder_to_record_key in GeneratePrompt transform

#301 xiaoyi-cheng closed 5 months ago
1
feat: add validate_prompt_template util

#300 xiaoyi-cheng closed 5 months ago
0
fix: Context precision score computation

#299 awsvmaringa closed 5 months ago
1
feat: faithfulness handle no statements edge case

#298 xiaoyi-cheng closed 5 months ago
0
feat: add error field in EvalScore

#297 xiaoyi-cheng closed 5 months ago
0
build(deps-dev): bump pdoc from 14.5.0 to 14.5.1

#296 dependabot[bot] closed 5 months ago
0
feat: add answer relevance algo

#295 xiaoyi-cheng closed 5 months ago
2
feat: add col_to_validate to evaluate_dataset

#294 oyangz closed 5 months ago
1
feat: support embedding model runner

#293 xiaoyi-cheng closed 5 months ago
0
build(deps): bump urllib3 from 1.26.18 to 1.26.19

#292 dependabot[bot] closed 5 months ago
2
feat: add faithfulness eval algo

#291 xiaoyi-cheng closed 5 months ago
0
merge main into rageval branch

#290 oyangz closed 5 months ago
1
feat: add context precision for context quality eval algo

#289 oyangz closed 5 months ago
2
feat: modify GeneratePrompt transform to take placeholder_dict

#288 xiaoyi-cheng closed 5 months ago
0
build(deps): bump tornado from 6.4 to 6.4.1

#287 dependabot[bot] closed 5 months ago
1
ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead.

#286 aakash086 closed 5 months ago
4