Timeout context manager should be used inside a task - botbuilder.ai.qna - qnaMaker.get_answers

[vijaysaimutyala]

Version

botbuilder.ai 4.9

Describe the bug

Timeout context manager should be used inside a task when using QnAMaker.get_answers

To Reproduce

Steps to reproduce the behavior: Below is my code. I check for the intent of a text from user and if it fails, I try to get the answer from QnA Maker. This is resulting in a timeout error.

from botbuilder.core import ActivityHandler, TurnContext, MessageFactory, IntentScore
from botbuilder.schema import ChannelAccount,Attachment
from botbuilder.ai.luis import LuisApplication,LuisPredictionOptions,LuisRecognizer
from botbuilder.ai.qna import QnAMakerEndpoint,QnAMakerOptions,QnAMaker
from typing import List
import os
import json
import os.path
from usecases import SimpleUseCase

class Bot(ActivityHandler):

    def __init__(self):
        luisApp = LuisApplication("","","")
        luisOptions = LuisPredictionOptions(include_all_intents=True,include_instance_data=True)
        self.LuisRecog = LuisRecognizer(luisApp,luisOptions)
        qna_endpoint = QnAMakerEndpoint("","","")
        qna_options = QnAMakerOptions(score_threshold=0.5)
        self.qnaMaker = QnAMaker(qna_endpoint)

    async def on_members_added_activity(
        self, members_added: List[ChannelAccount], turn_context: TurnContext
    ):
        for member in members_added:
            # Greet anyone that was not the target (recipient) of this message.
            # To learn more about Adaptive Cards, see https://aka.ms/msbot-adaptivecards for more details.
            if member.id != turn_context.activity.recipient.id:
                welcome_card = self.create_adaptive_card_attachment()
                response = MessageFactory.attachment(welcome_card)
                await turn_context.send_activity(response)

    # Load attachment from file.
    def create_adaptive_card_attachment(self):
        relative_path = os.path.abspath(os.path.dirname(__file__))
        path = os.path.join(relative_path, "cards/welcomeCard.json")
        with open(path) as in_file:
            card = json.load(in_file)

        return Attachment(
            content_type="application/vnd.microsoft.card.adaptive", content=card
        )

    async def greetings(self,turn_context):
        await turn_context.send_activity(MessageFactory.text("from inside greetings"))

    async def on_message_activity(self, turn_context: TurnContext):
        luisResult = await self.LuisRecog.recognize(turn_context) 
        print(luisResult.get_top_scoring_intent())
        intent = LuisRecognizer.top_intent(luisResult,min_score=0.40)

        if turn_context.activity.text.lower() == 'hi':
            await self.greetings(turn_context)
            await turn_context.send_activity(MessageFactory.text("message sent from send greetings.."))
        else:
            if intent != "None":
                response = await SimpleUseCase().someUseCase(turn_context)
                await turn_context.send_activity(response)
            else:
                await turn_context.send_activity(MessageFactory.text("Did not finda any use cases..checking QnA Maker..."))
                answers = await self.qnaMaker.get_answers(turn_context)
                print(answers)
                await turn_context.send_activity(MessageFactory.text(answers[0].answer))     

from quart import Quart, request, Response
from botbuilder.core import (
    BotFrameworkAdapterSettings,
    ConversationState,
    MemoryStorage,
    UserState,
)
from botbuilder.schema import Activity

from bot import Bot

from adapter_with_error_handler import AdapterWithErrorHandler

app = Quart(__name__, instance_relative_config=True)
app.config.from_object("config.DefaultConfig")

# Create adapter.
# See https://aka.ms/about-bot-adapter to learn more about how bots work.
SETTINGS = BotFrameworkAdapterSettings(app.config["APP_ID"], app.config["APP_PASSWORD"])

# Create MemoryStorage, UserState and ConversationState
MEMORY = MemoryStorage()
USER_STATE = UserState(MEMORY)
CONVERSATION_STATE = ConversationState(MEMORY)

# Create adapter.
# See https://aka.ms/about-bot-adapter to learn more about how bots work.
ADAPTER = AdapterWithErrorHandler(SETTINGS, CONVERSATION_STATE)

# Create dialogs and Bot
BOT = Bot()

# Listen for incoming requests on /api/messages
@app.route("/api/messages", methods=["POST"])
async def messages():
    # Main bot message handler.
    if "application/json" in request.headers["Content-Type"]:
        body = await request.json
    else:
        return Response("", status=415)

    activity = Activity().deserialize(body)
    auth_header = (
        request.headers["Authorization"] if "Authorization" in request.headers else ""
    )

    try:
        await ADAPTER.process_activity(activity, auth_header, BOT.on_turn)
        return Response("", status=201)
    except Exception as exception:
        raise exception

@app.route("/", methods=["GET"])
async def homepage():
    try:
        return "Yes I'm working brother."
    except Exception as exception:
        raise exception

if __name__ == "__main__":
    try:
        app.run()  # nosec debug
    except Exception as exception:
        raise exception

Expected behavior

I should be receiving answer from QnAMaker. I've tried the QnAMaker component individually and it worked well without this issue

Screenshots

Additional context

I'm using the Quart samples to build the bot. I've tried using the REST API for QnAMaker with the requests module and it worked without any issues. Also, I ran this with Flask and asyncio and that too worked without any issues.

[vijaysaimutyala]

A quick update on this one. I've tried by creating the QnAMaker object dynamically while processing and this worked without any issues. I'm a bit new to async and not sure what's happening.

    async def on_message_activity(self, turn_context: TurnContext):
        luisResult = await self.LuisRecog.recognize(turn_context) 
        print(luisResult.get_top_scoring_intent())
        intent = LuisRecognizer.top_intent(luisResult,min_score=0.40)

        if turn_context.activity.text.lower() == 'hi':
            await self.greetings(turn_context)
            await turn_context.send_activity(MessageFactory.text("message sent from send greetings.."))
        else:
            if intent != "None":
                response = await SimpleUseCase().someUseCase(turn_context)
                await turn_context.send_activity(response)
            else:
                await turn_context.send_activity(MessageFactory.text("Did not finda any use cases..checking QnA Maker..."))
                answers = await QnAMaker(QnAMakerEndpoint("","","")).get_answers(turn_context)                
                await turn_context.send_activity(MessageFactory.text(answers[0].answer))

[scheyal]

@axelsrz rz, please look at this.

[axelsrz]

Hello @vijaysaimutyala thanks for loging this. We will try to replicate this is the minimal necessary elements and investigate if there is an incompatibility between the qna lib and quart on our next cycle.

[compulim]

@axelsrz It is still in our radar, right?

[Zerryth]

Hey William,

Yes, still on our radar! Just slating it more closely to next milestone rather than R10, given team bandwidth. If time, will try to get to this sooner than that

[Zerryth]

Doesn't look like we'll have time to get to this during the R10 timeframe, and will fall under R11 instead

[daveta]

Hi @Zerryth : Thanks for update.

[Zerryth]

@vijaysaimutyala Just tried running 14.nlp-with-dispatch sample that uses both QnAMaker & LUIS as you have, including creating a QnAMaker service upon initialization of the bot instance and saving it as a member (as opposed to dynamically creating a new QnA service every single time when you want to make a query), and it works just fine as expected. See Use multiple LUIS and QnA models for more details on how to set up that sample yourself, if interested.

Haven't tried using QnA in a Quart project yet, so I'll get back to you after looking into Quart itself, to see if I can actually get a repro of your issue.

[Zerryth]

Just wanted to update the thread to say that I tried opening python_quart/13.core-bot, plugged in calling QnAMaker if it didn't have an intent from LUIS, and was able to reproduce @vijaysaimutyala 's error:

app.py

# Create dialogs and Bot
LUIS_RECOGNIZER = FlightBookingRecognizer(APP.config)
QNAMAKER = QnAMaker(
    QnAMakerEndpoint(
        knowledge_base_id=APP.config["QNA_KNOWLEDGEBASE_ID"],
        endpoint_key=APP.config["QNA_ENDPOINT_KEY"],
        host=APP.config["QNA_ENDPOINT_HOST"],
    )
)
BOOKING_DIALOG = BookingDialog()
DIALOG = MainDialog(LUIS_RECOGNIZER, QNAMAKER, BOOKING_DIALOG)
BOT = DialogAndWelcomeBot(CONVERSATION_STATE, USER_STATE, DIALOG)

main_dialog.py

# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.

from botbuilder.dialogs import (
    ComponentDialog,
    WaterfallDialog,
    WaterfallStepContext,
    DialogTurnResult,
)
from botbuilder.ai.qna import QnAMaker
from botbuilder.dialogs.prompts import TextPrompt, PromptOptions
from botbuilder.core import MessageFactory, TurnContext
from botbuilder.schema import InputHints

from booking_details import BookingDetails
from flight_booking_recognizer import FlightBookingRecognizer
from helpers.luis_helper import LuisHelper, Intent
from .booking_dialog import BookingDialog

class MainDialog(ComponentDialog):
    def __init__(
        self, luis_recognizer: FlightBookingRecognizer, qnamaker: QnAMaker, booking_dialog: BookingDialog
    ):
        super(MainDialog, self).__init__(MainDialog.__name__)

        self._luis_recognizer = luis_recognizer
        self._qnamaker = qnamaker
        self._booking_dialog_id = booking_dialog.id

        self.add_dialog(TextPrompt(TextPrompt.__name__))
        self.add_dialog(booking_dialog)
        self.add_dialog(
            WaterfallDialog(
                "WFDialog", [self.intro_step, self.act_step, self.final_step]
            )
        )

        self.initial_dialog_id = "WFDialog"

    async def intro_step(self, step_context: WaterfallStepContext) -> DialogTurnResult:
        if not self._luis_recognizer.is_configured:
            await step_context.context.send_activity(
                MessageFactory.text(
                    "NOTE: LUIS is not configured. To enable all capabilities, add 'LuisAppId', 'LuisAPIKey' and "
                    "'LuisAPIHostName' to the appsettings.json file.",
                    input_hint=InputHints.ignoring_input,
                )
            )

            return await step_context.next(None)
        message_text = (
            str(step_context.options)
            if step_context.options
            else "What can I help you with today?"
        )
        prompt_message = MessageFactory.text(
            message_text, message_text, InputHints.expecting_input
        )

        return await step_context.prompt(
            TextPrompt.__name__, PromptOptions(prompt=prompt_message)
        )

    async def act_step(self, step_context: WaterfallStepContext) -> DialogTurnResult:
        if not self._luis_recognizer.is_configured:
            # LUIS is not configured, we just run the BookingDialog path with an empty BookingDetailsInstance.
            return await step_context.begin_dialog(
                self._booking_dialog_id, BookingDetails()
            )

        # Call LUIS and gather any potential booking details. (Note the TurnContext has the response to the prompt.)
        intent, luis_result = await LuisHelper.execute_luis_query(
            self._luis_recognizer, step_context.context
        )

        if intent == Intent.BOOK_FLIGHT.value and luis_result:
            # Show a warning for Origin and Destination if we can't resolve them.
            await MainDialog._show_warning_for_unsupported_cities(
                step_context.context, luis_result
            )

            # Run the BookingDialog giving it whatever details we have from the LUIS call.
            return await step_context.begin_dialog(self._booking_dialog_id, luis_result)

        if intent == Intent.GET_WEATHER.value:
            get_weather_text = "TODO: get weather flow here"
            get_weather_message = MessageFactory.text(
                get_weather_text, get_weather_text, InputHints.ignoring_input
            )
            await step_context.context.send_activity(get_weather_message)

        else:
            calling_qna_text = (
                "No LUIS intent recognized. Calling QnAMaker..."
            )
            calling_qna_message = MessageFactory.text(
                calling_qna_text, calling_qna_text, InputHints.ignoring_input
            )
            await step_context.context.send_activity(calling_qna_message)

            await self._call_qnamaker(step_context.context)

        return await step_context.next(None)

    async def final_step(self, step_context: WaterfallStepContext) -> DialogTurnResult:
        # If the child dialog ("BookingDialog") was cancelled or the user failed to confirm,
        # the Result here will be null.
        if step_context.result is not None:
            result = step_context.result

            # Now we have all the booking details call the booking service.

            # If the call to the booking service was successful tell the user.
            # time_property = Timex(result.travel_date)
            # travel_date_msg = time_property.to_natural_language(datetime.now())
            msg_txt = f"I have you booked to {result.destination} from {result.origin} on {result.travel_date}"
            message = MessageFactory.text(msg_txt, msg_txt, InputHints.ignoring_input)
            await step_context.context.send_activity(message)

        prompt_message = "What else can I do for you?"
        return await step_context.replace_dialog(self.id, prompt_message)

    async def _call_qnamaker(self, turn_context: TurnContext):
        results = await self._qnamaker.get_answers(turn_context)

        if results:
            await turn_context.send_activity(results[0].answer)
        else:
            await turn_context.send_activity(
                "Sorry, could not find an answer in the Q and A system."
            )

    @staticmethod
    async def _show_warning_for_unsupported_cities(
        context: TurnContext, luis_result: BookingDetails
    ) -> None:
        if luis_result.unsupported_airports:
            message_text = (
                f"Sorry but the following airports are not supported:"
                f" {', '.join(luis_result.unsupported_airports)}"
            )
            message = MessageFactory.text(
                message_text, message_text, InputHints.ignoring_input
            )
            await context.send_activity(message)

[on_turn_error] unhandled error: Timeout context manager should be used inside a task
Traceback (most recent call last):
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\core\bot_adapter.py", line 127, in run_pipeline
    return await self._middleware.receive_activity_with_status(
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\core\middleware_set.py", line 69, in receive_activity_with_status
    return await self.receive_activity_internal(context, callback)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\core\middleware_set.py", line 79, in receive_activity_internal
    return await callback(context)
  File "c:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\bots\dialog_bot.py", line 30, in on_turn   
    await super().on_turn(turn_context)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\core\activity_handler.py", line 68, in on_turn
    await self.on_message_activity(turn_context)
  File "c:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\bots\dialog_bot.py", line 37, in on_message_activity
    await DialogHelper.run_dialog(
  File "c:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\helpers\dialog_helper.py", line 17, in run_dialog
    results = await dialog_context.continue_dialog()
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\dialog_context.py", line 128, in continue_dialog
    return await dialog.continue_dialog(self)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\component_dialog.py", line 105, in continue_dialog
    turn_result = await self.on_continue_dialog(inner_dc)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\component_dialog.py", line 230, in on_continue_dialog
    return await inner_dc.continue_dialog()
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\dialog_context.py", line 128, in continue_dialog
    return await dialog.continue_dialog(self)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\prompts\prompt.py", line 146, in continue_dialog
    return await dialog_context.end_dialog(recognized.value)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\dialog_context.py", line 158, in end_dialog
    return await dialog.resume_dialog(self, DialogReason.EndCalled, result)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\waterfall_dialog.py", line 97, in resume_dialog
    return await self.run_step(
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\waterfall_dialog.py", line 156, in run_step
    return await self.on_step(step_context)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\dialogs\waterfall_dialog.py", line 132, in on_step
    return await self._steps[step_context.index](step_context)
  File "c:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\dialogs\main_dialog.py", line 110, in act_step
    await self._call_qnamaker(step_context.context)
  File "c:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\dialogs\main_dialog.py", line 133, in _call_qnamaker
    results = await self._qnamaker.get_answers(turn_context)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\ai\qna\qnamaker.py", line 87, in get_answers
    result = await self.get_answers_raw(
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\ai\qna\qnamaker.py", line 114, in get_answers_raw
    result = await self._generate_answer_helper.get_answers_raw(context, options)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\ai\qna\utils\generate_answer_utils.py", line 81, in get_answers_raw
    result: QueryResults = await self._query_qna_service(context, hydrated_options)
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\ai\qna\utider\ai\qna\utils\generate_answer_utils.py", line 165, in _query_qna_service
    response: ClientResponse = await http_request_helper.execute_http_request(
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\botbuilder\ai\qna\utider\ai\qna\utils\http_request_utils.py", line 65, in execute_http_request
    response: ClientResponse = await self._http_client.post(
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\aiohttp\client.py", l\client.py", line 426, in _request
    with timer:
  File "C:\Users\AshleyFinafrock\Work\quart-sample\samples\python\wip\python_quart\13.core-bot\env\lib\site-packages\aiohttp\helpers.py", \helpers.py", line 579, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task
Datetime with no tzinfo will be considered UTC.
[2020-08-25 13:33:39,212] 127.0.0.1:63967 POST /api/messages 1.1 201 0 39651815

Additionally I also did get it to work when creating a new QnAMaker instance every time you wanted to call the GetAnswer API, just as vijaysaimutyala had done in his update comment as well--this is the pattern that's used anyways in sample 11.qnamaker for C# and can be used as the workaround for now.

Going to dig down and see if this is a Quart-specific issue by plugging in QnAMaker into the "regular" sample 13.core-bot and see if it also has the error or not

[Zerryth]

Alright, just plugged QnAMaker into the "regular" samples/python/13.core-bot, and it actually does work. There indeed does seem to be an incompatibility between Quart and the QnAMaker library.