dosubot[bot]
commented
5 months ago Hey there, @WhyMEliu! Fancy seeing you around these parts again. Hope all is well on your end! 🦙💻
To make HTTP POST requests to your LLM service and Embedding using LlamaIndex, you can utilize the CustomLLM class. This class has two methods, _complete and _stream_complete, which you can override to handle the specifics of making HTTP POST requests to your LLM service.
Here's an example of how you might implement these methods to make HTTP POST requests using the requests library. This example assumes your LLM service accepts JSON payloads and returns JSON responses.
First, ensure you have the requests library installed:
pip install requestsThen, modify the CustomLLM class by adding the _complete and _stream_complete methods:
import requests
from typing import Any, Dict
class CustomLLM(LLM):
def __init__(self, service_url: str):
self.service_url = service_url
def _complete(self, prompt: str, formatted: bool = False, **kwargs: Any) -> CompletionResponse:
# Construct the payload for the POST request
payload = {
"prompt": prompt,
"formatted": formatted,
**kwargs
}
# Make the HTTP POST request
response = requests.post(self.service_url, json=payload)
# Ensure the request was successful
response.raise_for_status()
# Parse the JSON response
data = response.json()
# Convert the response to a CompletionResponse (you may need to adjust this based on your service's response structure)
return CompletionResponse(
completions=data.get("completions"),
metadata=data.get("metadata")
)
def _stream_complete(self, prompt: str, formatted: bool = False, **kwargs: Any) -> CompletionResponseAsyncGen:
# This example assumes streaming is handled via successive POST requests; adjust based on your actual streaming implementation
# For real streaming support, you might need a different approach, possibly involving websockets or another streaming protocol
raise NotImplementedError("Streaming completions are not implemented in this example")
# Implement the `metadata` method as required by the abstract base class
def metadata(self) -> Dict[str, Any]:
return {
"service_url": self.service_url
}This example provides a basic implementation for making synchronous HTTP POST requests to an LLM service using the _complete method. The _stream_complete method is left unimplemented because streaming behavior can vary significantly between services, and some services might not support streaming completions via HTTP POST. You would need to adjust these implementations based on the specifics of your LLM service, including how it expects requests to be formatted and how it returns responses.
You can find more information about the CustomLLM class in the llama-index-core/llama_index/core/llms/custom.py file.
Additionally, LlamaIndex does provide built-in support for making HTTP requests. This is evident from the llama_index/tools/requests/base.py file, which shows a RequestsToolSpec class that wraps the functionality of the requests library for making GET, POST, and PATCH requests. This class is part of the LlamaIndex framework's integrations/tools, indicating that users do not need to use an external library like requests directly for these operations, as the framework already includes this capability. You can find more about this in the llama-index-integrations/tools/llama-index-tools-requests/llama_index/tools/requests/base.py file.
Please note that this is a basic example and you might need to adjust it based on the specifics of your LLM service and Embedding.
Sources
- llama-index-core/llama_index/core/llms/custom.py
- llama-index-integrations/tools/llama-index-tools-requests/llama_index/tools/requests/base.py
logan-markewich
WhyMEliu
xjw1001001
Question Validation
Question
I have an LLM service that exposes only http post requests. The same goes for Embedding. How do I use llamaindex to call them?