Extend Support: Mypy Results

[KushalGaywala]

Adding support for more mypy resultant errors would help in less feedback iterations for a completion.

I analyzed both level_1__ft__mypy_signature_5_steps__1_choice.yaml and level_1__ft__mypy_jedi_5_steps__1_choice.yaml and found that both were not able to find signatures for completion.create but while the integration process of both signature and jedi they were able to fetch signature for this method.

Here client.chat.completions.create results in error for wrong type of parameter. Hence, the result from analyzers should be the following:

create(*, messages: List[ChatCompletionMessageParam], model: Union[
            str,
            Literal[
                "gpt-4-1106-preview",
                "gpt-4-vision-preview",
                "gpt-4",
                "gpt-4-0314",
                "gpt-4-0613",
                "gpt-4-32k",
                "gpt-4-32k-0314",
                "gpt-4-32k-0613",
                "gpt-3.5-turbo",
                "gpt-3.5-turbo-16k",
                "gpt-3.5-turbo-0301",
                "gpt-3.5-turbo-0613",
                "gpt-3.5-turbo-16k-0613",
            ],
        ], frequency_penalty: Optional[float] | NotGiven=NOT_GIVEN, function_call: completion_create_params.FunctionCall | NotGiven=NOT_GIVEN, functions: List[completion_create_params.Function] | NotGiven=NOT_GIVEN, logit_bias: Optional[Dict[str, int]] | NotGiven=NOT_GIVEN, max_tokens: Optional[int] | NotGiven=NOT_GIVEN, n: Optional[int] | NotGiven=NOT_GIVEN, presence_penalty: Optional[float] | NotGiven=NOT_GIVEN, response_format: completion_create_params.ResponseFormat | NotGiven=NOT_GIVEN, seed: Optional[int] | NotGiven=NOT_GIVEN, stop: Union[Optional[str], List[str]] | NotGiven=NOT_GIVEN, stream: Optional[Literal[False]] | NotGiven=NOT_GIVEN, temperature: Optional[float] | NotGiven=NOT_GIVEN, tool_choice: ChatCompletionToolChoiceOptionParam | NotGiven=NOT_GIVEN, tools: List[ChatCompletionToolParam] | NotGiven=NOT_GIVEN, top_p: Optional[float] | NotGiven=NOT_GIVEN, user: str | NotGiven=NOT_GIVEN, extra_headers: Headers | None=None, extra_query: Query | None=None, extra_body: Body | None=None, timeout: float | httpx.Timeout | None | NotGiven=NOT_GIVEN) -> ChatCompletion

We need to get the line and the column of the method causing the error, and then pass to JediAnalyzer to get a specific signature that would really help and integrate functionality into SignatureAnalyzer, and see if it helps significantly.

We could also clean the output of the MypyAnalyzer and provide the results or just leave them as it is. What you guys think @psaegert @anacarsi . Currently I will start evaluation of the JediAnalyzer without solving these error to have some results.

Whole trace of Error:

while len(chat_completions) < N_SAMPLES:
    chat_completion = client.chat.completions.create(
model="gpt-3.5-turbo",
                                                    messages=messages)
    chat_completions.extend(chat_completion['choices'][0]['message']['content'])
    pbar.update(1)
[INST]
The completion you provided resulted in the following errors:
your completion:62: error: Argument "messages" to "create" of "Completions" has incompatible type "list[dict[str, str]]"; expected "list[ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam]"  [arg-type]
your completion:63: error: Value of type "ChatCompletion" is not indexable  [index]
Found 7 errors in 1 file (checked 1 source file)

[psaegert]

We could also clean the output of the MypyAnalyzer and provide the results or just leave them as it is

Sure, we could write a MypyResultsParser class that extracts the problematic classes, functions and attributes. This could then be used in both the SignatureAnalyzer and JediAnalyzer.

We need to get the line and the column of the method causing the error

Getting the column could be tricky, maybe we could match the problematic call in the code with regex?

found that both were not able to find signatures for completion.create but while the integration process of both signature and jedi they were able to fetch signature for this method.

I'm not sure if I understand this correctly. Did they not parse this line of the mypy results or just couldn't find the package and therefore no signatures? This line

your completion:62: error: Argument "messages" to "create" of "Completions" has incompatible type ...

should already be parsed in the current version of the SignatureAnalyzer:

# SignatureAnalyzer.py:353
patterns_attribute = {
    "has_attribute": re.compile(r'to \"(.+?)\" of \"(.+?)\" has'),
    ...