While I have tried FIM with the provider oobabooga, using the Fill-In-the-Middle models deepseek-coder-6.7b-instruct.Q8_0.gguf, DeepSeek-Coder-V2-Lite-Instruct, and mistralai/Codestral-22B-v0.1, none of them output good code, if at all.
I want to try other models without FIM (not just GPT-4o). To do so, I propose adding a simple Q&A mode.
This could also use gpt-4o from OpenAI API or any OpenAI-compatible endpoint: to demonstrate free GitHub model.
```python
import os, requests
# This requires GITHUB_TOKEN and 'set OPENAI_API_BASE=https://models.inference.ai.azure.com/chat/completions'
def chat(prompt, api_key=None, model=None, base_url=None, temp=1.0, top_p=1.0, max_tokens=1000, system_prompt="You are a helpful assistant."):
try:
response = requests.post(
base_url,
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
json={
"model": model,
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": prompt}
],
"temperature": temp,
"top_p": top_p,
"max_tokens": max_tokens
}
)
return response.json()["choices"][0]["message"]["content"].strip()
except Exception as e:
return f"Error: {str(e)}"
if __name__ == "__main__":
system_prompt = "You are a fill-in-the-middle code completion assistant. You only fill in by the missing code, do not modify or repeat the existing code."
prefix = "def calculate_area(radius):\n "
suffix = "\n return area"
prompt = f"{prefix}{suffix}"
model = 'gpt-4o'
answer = chat(prompt, api_key=os.getenv('GITHUB_TOKEN'), base_url=os.getenv('OPENAI_API_BASE'), model=model, system_prompt=system_prompt, max_tokens=100, temp=0.1) # CodeLlama uses the special token instead of <｜fim▁begin｜>, <｜fim▁hole｜> <｜fim▁end｜> but "# Fill in the code here" works as well.
print(f"{answer}")
```
This print:
```python
area = 3.14159 * radius * radius
```
While I have tried FIM with the provider oobabooga, using the Fill-In-the-Middle models
deepseek-coder-6.7b-instruct.Q8_0.gguf,DeepSeek-Coder-V2-Lite-Instruct, and mistralai/Codestral-22B-v0.1, none of them output good code, if at all.I want to try other models without FIM (not just GPT-4o). To do so, I propose adding a simple Q&A mode.
This could also use
```python import os, requests # This requires GITHUB_TOKEN and 'set OPENAI_API_BASE=https://models.inference.ai.azure.com/chat/completions' def chat(prompt, api_key=None, model=None, base_url=None, temp=1.0, top_p=1.0, max_tokens=1000, system_prompt="You are a helpful assistant."): try: response = requests.post( base_url, headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}, json={ "model": model, "messages": [ {"role": "system", "content": system_prompt}, {"role": "user", "content": prompt} ], "temperature": temp, "top_p": top_p, "max_tokens": max_tokens } ) return response.json()["choices"][0]["message"]["content"].strip() except Exception as e: return f"Error: {str(e)}" if __name__ == "__main__": system_prompt = "You are a fill-in-the-middle code completion assistant. You only fill ingpt-4ofrom OpenAI API or any OpenAI-compatible endpoint: to demonstrate free GitHub model.