huggingface / text-generation-inference

Large Language Model Text Generation Inference

http://hf.co/docs/text-generation-inference

Apache License 2.0

8.48k stars 968 forks source link

Unable to use '--rope-scaling yarn' option #1649

Closed calycekr closed 3 months ago

calycekr commented 4 months ago

Bug Report

PR: Adding yarn support https://github.com/huggingface/text-generation-inference/pull/1099 We can find 'yarn' for rope_scaling type. elif rope_scaling["type"] == "yarn":

But, there is no yarn in possible values in the tgi launcher.

error: invalid value 'yarn' for '--rope-scaling <ROPE_SCALING>'
[possible values: linear, dynamic]

No yarn support in this file. https://github.com/huggingface/text-generation-inference/blob/23fba672e8156ce19cc518470a5452a9543c56b9/launcher/src/main.rs#L107-L119

1320

1382

System Info

Information

[X] Docker
[X] The CLI directly

Tasks

[ ] An officially supported command
[ ] My own modifications

Reproduction

Use '--rope-scaling yarn' option.

Expected behavior

Support yarn option.

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.