search

jxnl / instructor

structured outputs for llms
https://python.useinstructor.com/
MIT License
7.52k stars 600 forks source link

"Not the same number of function calls and responses" when using Mistral API #724

Open AndreaDiTrani opened 3 months ago

AndreaDiTrani commented 3 months ago

What Model are you using?

Describe the bug I have some documents regarding electrical components and i am trying to extract the relevant features from them. On some samples of those documents everything works fine but on some of them it fails with the following exception:

MistralAPIException: Status: 400. Message: {"object":"error","message":"Not the same number of function calls and responses","type":"invalid_request_error","param":null,"code":null}

To Reproduce The pydantic structure i am using is the following:

FieldNames = enum.Enum('FieldNames', {item: item for item in field_list})

class Feature(BaseModel):
    id:int = Field(default = 0, description="Unique ID of the feature")
    feature_tag:FieldNames = Field(default = FieldNames('Power dissipation'), # type: ignore
                                   description = "Simple tag describing the feature") 
    feature_val:str = Field(default = 'none', description="Value of the feature")

class ProductFeatures(BaseModel):
    product:str = Field(default = 'none', description='id of the product')
    features: List[Feature] = Field(default = [Feature()], description="List of features")

and this is the function i'm calling to extract the features

def feature_extractor(info) -> ProductFeatures:
    EXTRACTION_MODEL = "open-mixtral-8x22b"
    messages = [
        {
            "role": "system",
            "content": extraction_prompt,
        },
        {
            "role": "user",
            "content": f"Consider: {info}\n Extract the features.",
        },
    ]
    plan = (
        patched_chat.chat.completions.create(
            model=EXTRACTION_MODEL,
            response_model=ProductFeatures,
            messages=messages,
            temperature=0,
            max_tokens=4096,
        )
    )
    return plan

Expected behavior The expected behavior is something similar to what happens with "sane" text samples:

{'product': 'X1-IS-AI-02',
 'features': [{'id': 1,
   'feature_tag': <FieldNames.Power dissipation: 'Power dissipation'>,
   'feature_val': '0.65 W (passive Tx), 1.05 W (active Tx) @ 24 Vdc with 20 mA, 24 Vdc + 250 Ω output, typical'},
  {'id': 2,
   'feature_tag': <FieldNames.System Supply: 'System Supply'>,
   'feature_val': '24 Vdc nom (18 to 30 Vdc)'},
  {'id': 3,
   'feature_tag': <FieldNames.Current consumption: 'Current consumption'>,
   'feature_val': '23 mA @ 24 Vdc with 20 mA, typical'},
  {'id': 4,
   'feature_tag': <FieldNames.System Out: 'System Out'>,
   'feature_val': 'Current 4 to 20 mA. The module always sinks current and the Termination Board extends its use to source loops'},
  {'id': 5,
   'feature_tag': <FieldNames.Response time: 'Response time'>,
   'feature_val': '≤ 20 ms (4 to 20 mA or 20 to 4 mA)'},
  {'id': 6,
   'feature_tag': <FieldNames.Current limitation: 'Current limitation'>,
   'feature_val': '25.5 mA typical'},
  {'id': 7,
   'feature_tag': <FieldNames.Maximum load: 'Maximum load'>,
   'feature_val': '600 Ω'},
  {'id': 8,
   'feature_tag': <FieldNames.Maximum voltage: 'Maximum voltage'>,
   'feature_val': '30 V'},
  {'id': 9,
   'feature_tag': <FieldNames.Minimum voltage: 'Minimum voltage'>,
   'feature_val': '15 V (including series resistance voltage drop)'},
  {'id': 10,
   'feature_tag': <FieldNames.Field In: 'Field In'>,
   'feature_val': 'Current 4 to 20 mA, passive or active Tx. Passive Tx line voltage: 15 V minimum @ 20 mA. Active Tx voltage drop: 6.5 V maximum @ 20 mA'},
  {'id': 11,
   'feature_tag': <FieldNames.Fault: 'Fault'>,
   'feature_val': 'An out-of-range fault with configurable limits is available on the common fault line'},
  {'id': 12,
   'feature_tag': <FieldNames.Performance: 'Performance'>,
   'feature_val': 'Ref. Conditions: 24 V supply, 24 V + 250 Ω output, 23 ± 1 °C ambient temperature. Linearity accuracy: ≤ ± 20 µA. Calibration accuracy: ≤ ± 20 µA. Temp. influence: ≤ ± 2 µA/°C. Isolation: Field In/System Out 2.5 kV; Field In/System Supply 2.5 kV; System Out/System Supply 500 V'},
  {'id': 13,
   'feature_tag': <FieldNames.Environmental conditions: 'Environmental conditions'>,
   'feature_val': 'Operating temperature: temperature limits –40 to +70 °C. Storage temperature: temperature limits –45 to +80 °C'},
  {'id': 14,
   'feature_tag': <FieldNames.Safety description: 'Safety description'>,
   'feature_val': 'Associated apparatus and non-sparking electrical equipment'},
  {'id': 15,
   'feature_tag': <FieldNames.Mounting: 'Mounting'>,
   'feature_val': 'On custom Termination Board'},
  {'id': 16,
   'feature_tag': <FieldNames.Weight: 'Weight'>,
   'feature_val': 'about 55 g'},
  {'id': 17,
   'feature_tag': <FieldNames.Dimensions: 'Dimensions'>,
   'feature_val': 'Width 10 mm, Depth 80 mm, Height 120 mm'}]}

If needed i can provide the prompts i'm using and also the text i'm trying to extract

Screenshots If applicable, add screenshots to help explain your problem.

geraud-recarta commented 2 weeks ago

I have the exact same issue. Based on the logs, it seems it's linked to the retry function. Also, in the meantime mistralai lib has been updated to 1.0.2 which is not retro-compatible with instructor.

Thanks for the good work!

Traceback (most recent call last): [personal code and folders are redacted] File ".py", line 94, in extract return self.client.messages.create( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/instructor/client.py", line 93, in create return self.create_fn( ^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/openinference/instrumentation/instructor/_wrappers.py", line 129, in patched_new_func resp = new_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/instructor/patch.py", line 143, in new_create_sync response = retry_sync( ^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/instructor/retry.py", line 162, in retry_sync for attempt in max_retries: File "/.venv/lib/python3.12/site-packages/tenacity/init.py", line 443, in iter do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/tenacity/init.py", line 376, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/tenacity/init.py", line 418, in exc_check raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/tenacity/init.py", line 185, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "__/.venv/lib/python3.12/site-packages/instructor/retry.py", line 165, in retry_sync response = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/mistralai/client.py", line 223, in chat for response in single_response: File "__/.venv/lib/python3.12/site-packages/mistralai/client.py", line 148, in _request yield self._check_response(response) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.12/site-packages/mistralai/client.py", line 77, in _check_response self._check_response_status_codes(response) File "/.venv/lib/python3.12/site-packages/mistralai/client.py", line 62, in _check_response_status_codes raise MistralAPIException.from_response( mistralai.exceptions.MistralAPIException: Status: 400. Message: {"object":"error","message":"Not the same number of function calls and responses","type":"invalid_request_error","param":null,"code":null}