[Bug]: Chat with your data: bot prints result of custom api call, but doesn't realise it has already received the results and keeps waiting.

[Aemulation]

Language

JS

Version

Tried both 1.1.0 and 1.3.1

Description

We're trying to make a "chat with your data" using our custom api. This api returns documents related to a search query, the bot should use those results to e.g. make a summary.

After asking the bot to look something up, it prints the result of the call to our api, after which it prints in another message "Searching for information about...". So while it has already received the data, it doesn't realize it and keeps waiting. Following up with the command "summarize your findings" results in the message "I am still searching for information about...".

The api spec used in the reproduction steps:

openapi: 3.0.0
servers:
  - url: https://api.zeta-alpha.com/v0/service/
    description: Production server
info:
  description: |
    A search engine for AI literature.

    <SecurityDefinitions />
  version: "0"
  title: Zeta Alpha Public API
paths:
  /documents/search:
    post:
      tags:
        - documents
      summary: Search documents
      description: New version of Search API enabling the retrieval of documents in an index with arbitrary fields. The search can be restricted based on the provided arbitrary filters. The results are paginated. The content of the document is stored in the representations.text field.
      parameters: []
      operationId: document_search_post
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                retrieval_unit:
                  description: Whether the results are entire documents (`document`) or text passages within the documents (`chunk`).
                  type: string
                  enum:
                    - document
                    - chunk
                    - sentence
                query_string:
                  type: string
                  description: Query string
                  maxLength: 8192
                  default: ""
                  example: What is a CNN?
                page:
                  type: integer
                  minimum: 0
                  default: 1
                page_size:
                  type: integer
                  minimum: 1
                  default: 10
              required:
                - retrieval_unit
      responses:
        "200":
          description: List of zero or more hits
          content:
            application/json:
              schema:
                type: object
                properties:
                  retrieval_unit:
                    description: Whether the results are entire documents (`document`) or text passages within the documents (`chunk`).
                    type: string
                    enum:
                      - document
                      - chunk
                      - sentence
                  hits:
                    type: array
                    items:
                      type: object
                      required:
                        - _id
                        - document_id
                        - highlight
                        - score
                      properties:
                        title:
                          title: Title
                          type: string
                        authors:
                          type: array
                          items:
                            type: string
                          title: Authors
                        created_at:
                          format: date-time
                          title: Created At
                          type: string
                        last_updated_at:
                          format: date-time
                          title: Last Updated At
                          type: string
                        uri:
                          nullable: true
                          title: Uri
                          type: string
                        document_id:
                          title: Document Id
                          type: string
                        custom_metadata:
                          description: Custom metadata fields of the documents of the defined index
                          title: Custom Metadata
                          type: object
                          properties:
                            representations:
                              type: object
                              properties:
                                text:
                                  type: string
                        highlight:
                          type: string
                          example: We present quasi-<strong>recurrent</strong> <strong>neural</strong> <strong>networks</strong> for <strong>neural</strong> sequence modeling.<br />Quasi-<strong>recurrent</strong> <strong>neural</strong> <strong>networks</strong> are related to several such recently described models, especially the strongly-typed <strong>recurrent</strong> <strong>neural</strong> <strong>networks</strong> (T-RNN)<br /><strong>Recurrent</strong> <strong>neural</strong> <strong>network</strong> based language model.
                required:
                  - retrieval_unit
                  - number_of_pages
                  - total_hits
                  - hits
        "400":
          description: Invalid inputs.
          content:
            application/json:
              schema:
                type: object
                required:
                  - message
                properties:
                  message:
                    type: string
        "404":
          description: Object not found.
          content:
            application/json:
              schema:
                type: object
                required:
                  - message
                properties:
                  message:
                    type: string
        "408":
          description: Request timed out.
          content:
            application/json:
              schema:
                type: object
                required:
                  - message
                properties:
                  message:
                    type: string
components:
  responses:
    Unauthorized:
      description: Unauthorized request.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string
    BadRequest:
      description: Invalid inputs.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string
    NotFound:
      description: Object not found.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string
    TooManyRequests:
      description: Too many requests. Try again later.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string
    Timeout:
      description: Request timed out.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string
    ServerError:
      description: Server Error.
      content:
        application/json:
          schema:
            type: object
            required:
              - message
            properties:
              message:
                type: string

I modified /src/app/app.ts as described in the reproduction steps Line 57-59

    const result = await path.post(parameter.path, parameter.body, {
      params: parameter.query,
    });

Has been changed to

    const params =
    {
      "retrieval_unit": "document",
      "query_string": parameter.query,
      "page": 1,
      "page_size": 10,
    };

    const result = await path.post(parameter.path, params, { params: {} });

Reproduction Steps

1. In vscode teams-toolkit extension: click
    create new app
    custom copilot
    chat with your data
    custom api
    --use the api spec described in the description
    select POST/documents/search
    typescript
    OpenAI
    insert personal api key
2. I slightly modified `/src/app/app.ts` to work with our api, see the description.
3. Click "Preview you teams app", select debug, which opens a window in the browser.
4. Type "Give me information about robot soccer with nao robots", after which it should print the results of the call to our api and the message "Searching for information about robot soccer with Nao robots..."

[garrytrinder]

I spoke with @Aemulation on Friday and I have been able to replicate using Teams AI library v1.3.1.

I noticed that

• the response from the API is not included in the chat response output, but is shown in the UI.
• after the initial response, when I send another prompt I get the following error:

{
    "role": "user",
    "content": "No arguments were sent with called action. Call the \"document_search_post\" action with required arguments as a valid JSON object."
  },

I have seen this error before in the .NET library, the first time the action is called it works, but any action called after the initial response returns this error, see my comment on #1130.

@MuyangAmigo the scaffolded project from TTK uses an older version of Teams AI library. Are there any plans to update the template to the latest version?

[lilyydu]

Hey @Aemulation, thanks for sharing this bug. I will need to repro this on my end to debug. In the meantime, @garrytrinder what was the response shown in the UI?

And what is the other prompt you sent that returned the error snippet?

Just to double check- this is with Python? I'm trying to follow TTK flow but only JS/TS is popping up

[garrytrinder]

what was the response shown in the UI?

These are screenshots taken minutes ago. No errors but you can see the message that is returned "Searching for information about XXX" after both responses. Previously in my tests the second prompt would never return a result.

And what is the other prompt you sent that returned the error snippet?

I've just tried to repro the error but it seems to be working fine.

Below is from a different app but is representative of the error that I was seeing.

Just to double check- this is with Python? I'm trying to follow TTK flow but only JS/TS is popping up

This is JS/TS, not Python.

[lilyydu]

Hey @Aemulation ,

Just tried repro'd on my end, but it looks to be working. Placed a breakpoint and the action does get hit, and the card gets rendered. There's also a followup message after the query.

[Aemulation]

Hi @lilyydu,

I've tried multiple times, but I never get a "search result has been retrieved" message. These are all separate chats.

Is this correct/expected behaviour?

Adittionally, I sometimes get these weird errors

Between attempts, I completely stopped the "preview your teams app" and restarted it.

[lilyydu]

Hi @Aemulation,

I can't seem to repro'd your responses but my working screenshot above should be the general expected behaviour. There can be variability with LLMs in general, but our differences could be partially with the model used. Which model are you using? I am assuming you are using OpenAI?

Does the above highlight all the code changes you have made to the bot?

CC @corinagum could you try on your end to see if you can repro as well?

[Aemulation]

Hi @lilyydu,

I'm indeed using OpenAI, and the change I mentioned is the only one I made to the bot.

[lilyydu]

@Aemulation - what version are you using?

[Aemulation]

Which version of teams-ai? I tried both 1.1.0 (the default in teams toolkit) and 1.3.1, but got similar results.

[lilyydu]

@Aemulation - version for the LLM

[Aemulation]

Aha, I'm using gpt-3.5-turbo @lilyydu

[corinagum]

@SubbaReddi could you please distribute to your team? @lilyydu is unable to get a repro.

[SubbaReddi]

@corinagum , @lilyydu : I can reproduce the issue with the provided repro steps. Results are appearing from document search but finally displaying following message: "Searching for information about..."

Find attachment for issue recording and chat response details:

ChatResponse.txt

https://github.com/user-attachments/assets/95126528-90eb-4a48-ad10-4d0cd8f60b78

[corinagum]

Hey @therealjohn, just wondering if you've seen issues like this with the TTK sample before. I'm beginning investigation but thought it was worth asking :)

[corinagum]

A small update: I am actively investigating this issue and spent most of today debugging this. I don't have any full answers yet, only some small notes that don't resolve the main issue at hand (yet). I will resume tomorrow with aacebo's help.