[BUG]: LLM Agents Kafka pipeline is broken

[dagardner-nv]

Version

24.06

Which installation method(s) does this occur on?

Docker, Conda, Source

Describe the bug.

The pipeline has a few issues:

• Kafka brokers is hard-coded to "auto" this should be exposed as a flag
• Kafka input topic name should similarly be exposed as a flag
• Pipeline has gone out of sync with the simple_pipeline
• The _build_engine and _build_agent_executor methods should be shared with the simple pipeline
• processing a question results in an exception from the agent chain

Minimum reproducible example

python examples/llm/main.py agents kafka

Relevant log output

Click here to see error details

> Entering new AgentExecutor chain...
Error running agent: 'NoneType' object has no attribute 'create'
Traceback (most recent call last):
  File "/home/dagardner/work/tmp/mrel2406/morpheus/llm/nodes/langchain_agent_node.py", line 101, in _run_single
    return await self._agent_executor.arun(metadata=metadata, **kwargs)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/_api/deprecation.py", line 157, in awarning_emitting_wrapper
    return await wrapped(*args, **kwargs)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 651, in arun
    await self.acall(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/_api/deprecation.py", line 157, in awarning_emitting_wrapper
    return await wrapped(*args, **kwargs)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 428, in acall
    return await self.ainvoke(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 212, in ainvoke
    raise e
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 203, in ainvoke
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/agents/agent.py", line 1481, in _acall
    next_step_output = await self._atake_next_step(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/agents/agent.py", line 1275, in _atake_next_step
    [
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/agents/agent.py", line 1275, in 
    [
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/agents/agent.py", line 1303, in _aiter_next_step
    output = await self.agent.aplan(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/agents/agent.py", line 751, in aplan
    full_output = await self.llm_chain.apredict(callbacks=callbacks, **full_inputs)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/llm.py", line 310, in apredict
    return (await self.acall(kwargs, callbacks=callbacks))[self.output_key]
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/_api/deprecation.py", line 157, in awarning_emitting_wrapper
    return await wrapped(*args, **kwargs)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 428, in acall
    return await self.ainvoke(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 212, in ainvoke
    raise e
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/base.py", line 203, in ainvoke
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/llm.py", line 275, in _acall
    response = await self.agenerate([inputs], run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain/chains/llm.py", line 142, in agenerate
    return await self.llm.agenerate_prompt(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
    return await self.agenerate(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
    output = await self._agenerate_helper(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper
    raise e
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper
    await self._agenerate(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_community/llms/openai.py", line 1194, in _agenerate
    full_response = await acompletion_with_retry(
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/site-packages/langchain_community/llms/openai.py", line 133, in acompletion_with_retry
    return await llm.async_client.create(**kwargs)
AttributeError: 'NoneType' object has no attribute 'create'
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized

Thread 0x00007fb410ffe6c0 (most recent call first):
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/threading.py", line 953 in run
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/dagardner/work/conda/envs/mrel2406/lib/python3.10/threading.py", line 973 in _bootstrap

Full env printout

Click here to see environment details

 [Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• [X] I agree to follow Morpheus' Code of Conduct
• [X] I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

Once fixed, remove this pipeline from docs/source/extra_info/known_issues.md doc and remove the warning in examples/llm/agents/README.md