Fill-in-the-middle completion using the `suffix` argument is missing

[NightMachinery]

OpenAI's completion API has a suffix parameter that allows one to supply the context after the completion to enable fill-in-the-middle completions:

prompt = "def say_hello("
suffix = """):
  print('hi', name)"""

response = client.completions.create(
    model="text-davinci-003",
    prompt=prompt,
    suffix=suffix,
    max_tokens=10
)
# response.choices[0].text should be name

This is not supported by together API:

TypeError: Completions.create() got an unexpected keyword argument 'suffix'

[orangetin]

hi @NightMachinery , our API does not support suffix as a parameter. Can I ask how this is different from just passing in prompt = prompt + suffix ?

[NightMachinery]

@orangetin Some models support fill-in-the-middle (FIM, infilling). E.g.,

• Code Llama 2

  • Structured FIM API by Fireworks

    <PRE> {prompt_prefix} <SUF>{prompt_suffix} <MID>

• DeepSeek

  <｜fim▁begin｜>def quick_sort(arr):
  if len(arr) <= 1:
      return arr
  pivot = arr[0]
  left = []
  right = []
  <｜fim▁hole｜>
      if arr[i] < pivot:
          left.append(arr[i])
      else:
          right.append(arr[i])
  return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜>

• StarCoder

  <fim_prefix>before <fim_suffix> after<fim_middle>

The OpenAI way of doing FIM is passing the "suffix" as a suffix parameter to the legacy completion endpoint.

[NightMachinery]

Basically, normal completions only see the context before the completion. FIM also shows the context after the completion. For auto-complete models (like what Github Copilot does), showing the context after is vital.

Just appending the after context to the original prompt does not work.

Here is an example for better understanding. Suppose we want to complete <COMPLETE_HERE>:

print("My name<COMPLETE_HERE>)
# Name: Armin Hajat

openai_text_complete(
    model="gpt-3.5-turbo-instruct",
    prompt=r"""
    print("My name
    """,
    suffix=")\n# Name: Armin Hajat",
    max_tokens=100,
    temperature=0,
    # stop=["\n"],
)

 is Armin Hajat"

which when inserted instead of <COMPLETE_HERE> would become:

print("My name is Armin Hajat")
# Name: Armin Hajat

[orangetin]

thanks for the info! we will add this to our feature list