huggingface / llm-vscode

LLM powered development for VSCode
Apache License 2.0
1.24k stars 133 forks source link

middle token and end token in the wrong place #55

Closed hzhangxyz closed 1 year ago

hzhangxyz commented 1 year ago

Here is my output

INPUT to API: (with parameters {"max_new_tokens":60,"temperature":0.2,"do_sample":true,"top_p":0.95,"stop":["<|endoftext|>"]}) 
<fim_prefix># language: Python
def qsort(l: list[int]) -> list[int]:
    if 1 <= len(l):
        p = l[0]
        l1 = [x for x in l if x < p]
        l2 = [x for x in l if x == p]
        l3 = [x for x in l if x > p]
        return l1+l2+l3
    else:
        return <fim_suffix>

# Example for calling qsort

l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(qsort(l))

# Output: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
<fim_middle>

I did not change the setting of this extensions. It seems fim_middle ad fim_suffix should be exchanged. Here is the description about it in the setting:

String that is sent to server is in format: {startToken}{code above cursor}{middleToken}{code below cursor if isFillMode=true}{endToken}.