[Bug] calls to LLMRails works first tieme and hangs on subsequent calls

[Vesalon]

Env: v0.7.1 python 3.10 Ubuntu 20.04 on WSL2

I have the following guardrail app set up:

structre:

guardrail.py
├── config
│   ├── rails
│   │   ├── offtopic.co
│   └── config.yml

config.yml:

models:
  - type: main
    engine: openai
    model: gpt-4-1106-preview
  - type: embeddings
    engine: FastEmbed
    model: all-MiniLM-L6-v2

offtopic.co:

# Off-topic

define user ask about politics
  "What do you think about the government?"
  "Which party should I vote for?"

define user ask about stock market
  "Which stock should I invest in?"
  "Would this stock 10x over the next year?"

def bot refuse politics
  "boo, politics"

def bot finance
  "don't ask me how to help your portfolio, man"

def bot pass
  "question is on topic"

define flow politics
  user ask about politics
  bot refuse politics

define flow stock market
  user ask about stock market
  bot refuse finance

define flow passthrough
  user ...
  bot pass

guardrail.py:

import os
import logging

from nemoguardrails import LLMRails, RailsConfig

os.environ["OPENAI_API_KEY"] = "<API KEY>"
logging.basicConfig(level=logging.DEBUG)

dir_path = os.path.dirname(os.path.realpath(__file__))
config_path = os.path.join(dir_path, "config/")
assert os.path.exists(config_path)
config = RailsConfig.from_path(config_path)
rails = LLMRails(config)

after I run python -i guardrails.py, my first call to the guardrails (e.g rails.generate("Should I buy Target stock?")) works as expected, but when I try to make any subsequent call I see an exception message telling me the event loop is closed. It also doesn't error out it just hangs after that :

>>> rails.generate("what's your favorite cheese?")
DEBUG:asyncio:Using selector: EpollSelector
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'UtteranceUserActionFinished', 'final_transcript': "what's your favorite cheese?"}
INFO:nemoguardrails.flows.runtime:Event :: UtteranceUserActionFinished {'final_transcript': "what's your favorite cheese?"}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '0ae1817d-01b0-48d7-a44c-d020f31eef83', 'event_created_at': '2024-02-11T02:43:32.952668+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'UserMessage', 'text': '$user_message'}}, 'action_result_key': None, 'action_uid': '03886d4b-11e1-45bf-9a25-aac0c67ef43d', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '0ae1817d-01b0-48d7-a44c-d020f31eef83', 'event_created_at': '2024-02-11T02:43:32.952668+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'UserMessage', 'text': '$user_message'}}, 'action_result_key': None, 'action_uid': '03886d4b-11e1-45bf-9a25-aac0c67ef43d', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'UserMessage', 'uid': 'fb818656-80ad-437e-925c-3352b12b1ac7', 'event_created_at': '2024-02-11T02:43:32.953554+00:00', 'source_uid': 'NeMoGuardrails', 'text': "what's your favorite cheese?"}
INFO:nemoguardrails.flows.runtime:Event :: UserMessage {'uid': 'fb818656-80ad-437e-925c-3352b12b1ac7', 'event_created_at': '2024-02-11T02:43:32.953554+00:00', 'source_uid': 'NeMoGuardrails', 'text': "what's your favorite cheese?"}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'd64b5978-bc26-4bcc-8774-bbf5cc0b2f6d', 'event_created_at': '2024-02-11T02:43:32.954350+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '23d92b95-3ad2-4e89-8ce0-aacf3457fe8a', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': 'd64b5978-bc26-4bcc-8774-bbf5cc0b2f6d', 'event_created_at': '2024-02-11T02:43:32.954350+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '23d92b95-3ad2-4e89-8ce0-aacf3457fe8a', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: generate_user_intent
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_user_intent
INFO:nemoguardrails.actions.llm.generation:Phase 1: Generating user intent
INFO:nemoguardrails.logging.callbacks:Invocation Params :: {'model': 'gpt-4-1106-preview', 'model_name': 'gpt-4-1106-preview', 'stream': False, 'n': 1, 'temperature': 0.0, '_type': 'openai-chat', 'stop': None}
INFO:nemoguardrails.logging.callbacks:Prompt Messages :: 
User
"""
Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.
"""

# This is how a conversation between a user and the bot can go:
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "Tell me a bit about the history of NVIDIA."
  ask general question
bot response for general question
  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis Priem."
user "tell me more"
  request more information
bot provide more information
  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming industry. The company continued to innovate in the GPU space, releasing new products and expanding into other markets such as professional graphics, mobile devices, and artificial intelligence."
user "thanks"
  express appreciation
bot express appreciation and offer additional help
  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't hesitate to ask."

# This is how the user talks:
user "Would this stock 10x over the next year?"
  ask about stock market

user "Which stock should I invest in?"
  ask about stock market

user "What do you think about the government?"
  ask about politics

user "Which party should I vote for?"
  ask about politics

# This is the current conversation between the user and the bot:
# Choose intent from this list: ask about stock market, ask about politics
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "what's your favorite cheese?"

DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'user', 'content': '"""\nBelow is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.\n"""\n\n# This is how a conversation between a user and the bot can go:\nuser "Hello there!"\n  express greeting\nbot express greeting\n  "Hello! How can I assist you today?"\nuser "What can you do for me?"\n  ask about capabilities\nbot respond about capabilities\n  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."\nuser "Tell me a bit about the history of NVIDIA."\n  ask general question\nbot response for general question\n  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis Priem."\nuser "tell me more"\n  request more information\nbot provide more information\n  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 1999, NVIDIA released the GeForce 256, the world\'s first GPU, which was a major breakthrough for the gaming industry. The company continued to innovate in the GPU space, releasing new products and expanding into other markets such as professional graphics, mobile devices, and artificial intelligence."\nuser "thanks"\n  express appreciation\nbot express appreciation and offer additional help\n  "You\'re welcome. If you have any more questions or if there\'s anything else I can help you with, please don\'t hesitate to ask."\n\n\n# This is how the user talks:\nuser "Would this stock 10x over the next year?"\n  ask about stock market\n\nuser "Which stock should I invest in?"\n  ask about stock market\n\nuser "What do you think about the government?"\n  ask about politics\n\nuser "Which party should I vote for?"\n  ask about politics\n\n\n\n# This is the current conversation between the user and the bot:\n# Choose intent from this list: ask about stock market, ask about politics\nuser "Hello there!"\n  express greeting\nbot express greeting\n  "Hello! How can I assist you today?"\nuser "What can you do for me?"\n  ask about capabilities\nbot respond about capabilities\n  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."\nuser "what\'s your favorite cheese?"\n'}], 'model': 'gpt-4-1106-preview', 'n': 1, 'stream': False, 'temperature': 0.0}}
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.failed exception=RuntimeError('Event loop is closed')
DEBUG:openai._base_client:Encountered Exception
Traceback (most recent call last):
  File "/home/gfertt/.local/lib/python3.10/site-packages/openai/_base_client.py", line 1423, in _request
    response = await self._client.send(
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 234, in handle_async_request
    await self._close_expired_connections()
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 195, in _close_expired_connections
    await connection.aclose()
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 173, in aclose
    await self._connection.aclose()
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 253, in aclose
    await self._network_stream.aclose()
  File "/home/gfertt/.local/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
    await self._stream.aclose()
  File "/home/gfertt/.local/lib/python3.10/site-packages/anyio/streams/tls.py", line 193, in aclose
    await self.transport_stream.aclose()
  File "/home/gfertt/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1261, in aclose
    self._transport.close()
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 706, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
DEBUG:openai._base_client:1 retry left
INFO:openai._base_client:Retrying request to /chat/completions in 0.938958 seconds
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'user', 'content': '"""\nBelow is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.\n"""\n\n# This is how a conversation between a user and the bot can go:\nuser "Hello there!"\n  express greeting\nbot express greeting\n  "Hello! How can I assist you today?"\nuser "What can you do for me?"\n  ask about capabilities\nbot respond about capabilities\n  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."\nuser "Tell me a bit about the history of NVIDIA."\n  ask general question\nbot response for general question\n  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis Priem."\nuser "tell me more"\n  request more information\nbot provide more information\n  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 1999, NVIDIA released the GeForce 256, the world\'s first GPU, which was a major breakthrough for the gaming industry. The company continued to innovate in the GPU space, releasing new products and expanding into other markets such as professional graphics, mobile devices, and artificial intelligence."\nuser "thanks"\n  express appreciation\nbot express appreciation and offer additional help\n  "You\'re welcome. If you have any more questions or if there\'s anything else I can help you with, please don\'t hesitate to ask."\n\n\n# This is how the user talks:\nuser "Would this stock 10x over the next year?"\n  ask about stock market\n\nuser "Which stock should I invest in?"\n  ask about stock market\n\nuser "What do you think about the government?"\n  ask about politics\n\nuser "Which party should I vote for?"\n  ask about politics\n\n\n\n# This is the current conversation between the user and the bot:\n# Choose intent from this list: ask about stock market, ask about politics\nuser "Hello there!"\n  express greeting\nbot express greeting\n  "Hello! How can I assist you today?"\nuser "What can you do for me?"\n  ask about capabilities\nbot respond about capabilities\n  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."\nuser "what\'s your favorite cheese?"\n'}], 'model': 'gpt-4-1106-preview', 'n': 1, 'stream': False, 'temperature': 0.0}}
DEBUG:httpcore.connection:close.started
DEBUG:httpcore.connection:close.complete
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gfertt/.local/lib/python3.10/site-packages/nemoguardrails/rails/llm/llmrails.py", line 547, in generate
    return asyncio.run(
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1871, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.10/selectors.py", line 469, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt

[vaibhavl98]

I am also facing the same issue. Please let me know if there's any update on this :).

[wadoodba]

I am facing the same issue is there any update on this?

[drazvan]

Thanks for reporting this @vaibhavl98 @Vesalon @wadoodba! It seems that OpenAI was caching the asyncio event loop, and the LLMRails instance was creating a new one on every sync call. This is fixed by #336 . Will be included in 0.8.0.