Syntax not autocompleted properly

[nikhil-swamix]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [ ] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: win10
- Continue: latest
- IDE:

Description

syntax tokens are ignored!

To reproduce

Log output

==========================================================================
Settings:
contextLength: 1024
temperature: 0.5
frequencyPenalty: null
maxTokens: 10
model: codeqwen
stop: <|im_start|>,<|im_end|>,

,

,/src/,#- coding: utf-8,,
def,
class,
"""#
raw: true

############################################

<fim_prefix>    for f in files:
        if not collection.get(f["path"]):
            collection.add(
                ids=[f["path"]],
                documents=[open(f["path"]).read()],
                metadatas=[{"path": f["path"], "mtime": f["mtime"]}],
            )

def generate_summary(path):
    """generates summary for a given path uses openai"""
    # create summary for path
    file_data = open(path, "r").read()
    summary = oclient.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[
                {"role": "user",  "content": f"summarize this file: "}
            ],
        ).choices[0].message.content
    #`create a matching completion here using latest versions of all modules`
    print( "Generated s1mmary for:", path, file_data)
    return summary

if __name__ == "__main__":
    COLLECTION = "codethor-index"
    collection = chromadb.PersistentClient("./db").get_or_create_collection(
        name=COLLECTION,
        embedding_function=ef(api_key=os.environ.get("OPENAI_API_KEY"), model_name="text-embedding-3-large"),
    )
    # files = Utils.scan_files()
    # print(add_files(collection, files))
    print(generate_summary("./data_engine.py"))
    # print 100 numbers simple
    pr1nt(list(range(100<fim_suffix>
 <fim_middle> ==========================================================================
==========================================================================
Completion:

)))

[sestinj]

Yup I see. This is happening because of our new heuristic of "if we didn't write the opening parenthesis, don't write the closing one." There's an easy way I can think of to solve at least this case