search

NVIDIA / RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
Apache License 2.0
662 stars 44 forks source link

GPT-4-1106-preview #63

Closed yxgcsq closed 1 month ago

yxgcsq commented 1 month ago

GPT-4-1106-preview What are the scores for each of the 13 tasks in the 4k test

yxgcsq commented 1 month ago

image This is the score I measured

hsiehjackson commented 1 month ago
Tasks niah_single_1 niah_single_2 niah_single_3 niah_multikey_1 niah_multikey_2 niah_multikey_3 niah_multivalue niah_multiquery vt fwe cwe qa_1 qa_2
Score 100.0 100.0 100.0 100.0 100.0 100.0 99.5 100.0 100.0 98.0 100.0 88.0 70.0

These are my results in the 4k test. Can you check the responses in variable tracking and common word extraction? I think maybe the API returns something unexpected.

yxgcsq commented 1 month ago

Thank you thank you