jlewitt1
commented
4 months ago Thanks for reaching out!
The FastAPI app you have set up here is probably not needed for this particular use case of deploying and calling a module (but do let us know otherwise if there's something specific you had in mind). Behind the scenes Runhouse spins up a FastAPI server on the cluster which will allow you to directly call the module on the cluster via an HTTP call. What you are seeing are the logs of the Runhouse server stored on the cluster in path: ~/.rh/server.log.
The below snippet should likely be all you need to deploy and call this module:
import runhouse as rh
import numpy as np
from scipy.special import softmax
from transformers import AutoModelForSequenceClassification, AutoConfig
from transformers import AutoTokenizer
class SentimentAnalysis:
def __init__(self, model_name="cardiffnlp/twitter-roberta-base-sentiment-latest"):
self.model_name = model_name
self.model = None
self.config = None
self.tokenizer = None
@staticmethod
def preprocess(text):
"""
Preprocess text (username and link placeholders)
"""
new_text = []
for t in text.split(" "):
t = '@user' if t.startswith('@') and len(t) > 1 else t
t = 'http' if t.startswith('http') else t
new_text.append(t)
return " ".join(new_text)
def predict(self, text):
if self.model is None:
self.model = AutoModelForSequenceClassification.from_pretrained(self.model_name)
if self.config is None:
self.config = AutoConfig.from_pretrained(self.model)
if self.tokenizer is None:
self.tokenizer = AutoTokenizer.from_pretrained()
text = SentimentAnalysis.preprocess(text)
encoded_input = self.tokenizer(text, return_tensors='pt')
output = self.model(**encoded_input)
scores = output[0][0].detach().numpy()
scores = softmax(scores)
ranking = np.argsort(scores)
ranking = ranking[::-1]
l_scores = {}
for i in range(scores.shape[0]):
l = self.config.id2label[ranking[i]]
s = scores[ranking[i]]
l_scores[l] = np.round(float(s), 4)
return l_scores
cluster = rh.ondemand_cluster(
name="fastapi-runhouse-example",
instance_type="CPU:2+",
provider="aws",
region="us-east-1"
).up_if_not()
# Set up an env using the requirements specified in the working dir
my_env = rh.env(name="scorer_env", working_dir="./")
# Send the module and its associated env to the cluster (or reload if it already exists on the cluster)
RemoteScorer = rh.module(SentimentAnalysis, name="remote-scorer").to(cluster, env=my_env)
# Generate a URL that can be used to call the module's "predict" method from anywhere
base_url = f"{RemoteScorer.endpoint()}/predict"
print(base_url)For example, to then call the predict method via CURL:
curl http://<CLUSTER-IP>:32300/remote-scorer/predict?text="Some text"
A couple other things to note based on the above snippet:
- We should generally specify an env for this module in order to ensure that the necessary requirements get installed on the cluster.
- Naming the module will make it more convenient to call later on - this can be specified in the module factory
- You can use
get_or_toinstead oftowhen sending the module to the cluster. This allows you to load the existing module by its name if it was already previously saved on the cluster - You can ssh onto the cluster using:
ssh fastapi-runhouse-exampleand restart the runhouse server anytime by runningrunhouse restarton the cluster - By setting the model, tokenizer, and config to None in the constructor of the
SentimentAnalysismodule we can prevent unnecessary reloading when hitting the “predict” endpoint
fryz
dongreenberg
Describe the bug
Example code: https://github.com/fryz/funhouse/tree/zf/fastapi/fastapi
When utilizing FastAPIs Lifespan Events (asynccontextmanager) to bring a cluster up, the cluster comes up but then hangs without returning to back to the server initialization logic.
Terminating the FastAPI process and bringing it back online recognizes the cluster and works.
Versions
Additional context
Logs from the startup: