how to set options for ollama?

[traddo]

client = instructor.from_openai(
    OpenAI(
        base_url="http://localhost:11434/v1",
        api_key="ollama",  # required, but unused
    ),
    mode=instructor.Mode.JSON,
)

def ask_ai(content) -> DocumentExtraction:
    return client.chat.completions.create(
        model="qwen:32b",
        response_model=DocumentExtraction,
        messages=[
            {
                "role": "system",
                "content": "Extract and resolve a list of entities from the following document:",
            },
            {
                "role": "user",
                "content": content,
            },
        ],
        options={
            "num_gpu": 100,
            "num_ctx": 32768,
            "temperature": 0,
            "top_k": 0,
            "top_p": 0,
        },
        stream=False,
    )

I'm encountering an error when trying to set parameters such as num_ctx and temperature for my request.

Traceback (most recent call last):
  File "/home/steven/work/dub_studio/a.py", line 74, in <module>
    resp = ask_ai(content)
  File "/home/steven/work/dub_studio/a.py", line 47, in ask_ai
    return client.chat.completions.create(
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/instructor/client.py", line 91, in create
    return self.create_fn(
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/instructor/patch.py", line 143, in new_create_sync
    response = retry_sync(
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/instructor/retry.py", line 162, in retry_sync
    for attempt in max_retries:
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
  File "/opt/conda/envs/gptr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/opt/conda/envs/gptr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/instructor/retry.py", line 165, in retry_sync
    response = func(*args, **kwargs)
  File "/opt/conda/envs/gptr/lib/python3.10/site-packages/openai/_utils/_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'options'

[vroomfondel]

It seems, this is not possible from the side of ollama mimicing the openai-api since the openai-api does not honor options even if you present them to it.

ollama doc on that: https://github.com/ollama/ollama/blob/main/docs/openai.md#setting-the-context-size

anyhow, if you want to include options in the request, this works: https://gist.github.com/vroomfondel/eb1fb4ac3319b22f9dc3d9ff658ce0c9