Exposing docstrings of `handler_predict` method on API docs in when using custom handlers for `predict`

[krumeto]

Is your feature request related to a problem? Please describe.

Vetiver API (and FastAPI) nicely display the docstrings of the function used to produce a given endpoint as a documentation on the API docs. However, when overwriting the predict method with a custom one, I have troubles making the docstring of the predict_handler available as docs.

For example, I have the following handler that is used as a basis for my predict and I'd like the docstring of handler_predict to be available as docs in the API docs:

from vetiver.handlers.base import BaseHandler
from vetiver import VetiverModel
from pydantic import BaseModel, Field, constr
from vetiver.prototype import vetiver_create_prototype
from pydantic.main import ModelMetaclass
import pandas as pd
import numpy as np

# Define a Pydantic model to validate the request body data
class RecommendationsRequest(BaseModel):
    param_1: constr(min_length=36, max_length=36, strip_whitespace=True)
    param_2: int = Field(gt=0,  default=5)
    param_3: int = Field(gt=0,  default=60)
    param_4: int = Field(gt=0,  default=1)

@vetiver_create_prototype.register
def _(data: ModelMetaclass):
    return data

class CustomHandler(BaseHandler):
    """A handler to wrap the predict method required by vetiver."""

    def __init__(self, model, prototype_data = RecommendationsRequest):

        super().__init__(model, prototype_data)
        model_type = staticmethod(lambda: 'custom handler')

    def handler_predict(self, request: RecommendationsRequest, check_prototype):
        """
        A method to overwrite the default `predict` method of the Vetiver BaseHandler

        Here follows a documentation that I'd like to see in the API docs.
        """

        if isinstance(request, list):
            param_1 = request[0]
            param_2 = request[1]
            param_3 = request[2]
            param_4 = request[3]
        else:
            param_1 = request.loc[0, 'param_1']
            # The item() below because otherwise it returns numpy.int64 instead of python ints
            # All numerics need that
            param_2 = request.loc[0, 'param_2'].item()
            param_3 = request.loc[0, 'param_3'].item()
            param_4 = request.loc[0, 'param_4'].item()

        prediction = self.model.get_article_recommendations(param_1=param_1,
                                                                      param_2=param_2,
                                                                      param_3=param_3,
                                                                      param_4=param_4)

        try:
            results = pd.Series(prediction['results'][0]) # numpy array, because Vetiver requires the results to have a `tolist()` method

        #FIXME Improve response when things are wrong
        except AttributeError:
            results = np.array([None])
        return results

Describe the solution you'd like I'd like the docstrings of the handler_predict method to be viewable in the API docs.

Thank you in advance!

[isabelizimm]

Hi there! Thanks for the report and try out vetiver 😄

Are you looking for docstrings in the API itself? I am unsure if those are exposed naturally, but could certainly look at what updates would be needed to make that happen. It would likely be an update to the open API specification.

One place where some documentation is rendered is at the top of the API, like the "A spacy English model" in the example below. This is actually the model's description, and you can set it when you create the VetiverModel object (ie, VetiverModel(model, "model_name", description = "anything you would like").

You can override the describe method if you would like to programmatically make it the handler_predict() docstring every time you use your custom handler. That would look something like this ⏬

class CustomHandler(BaseHandler):
    """A handler to wrap the predict method required by vetiver."""

    def __init__(self, model, prototype_data = RecommendationsRequest):

    ...

    def describe(self):

        desc = self.handler_predict.__doc__
        return desc

    def handler_predict(self, request: RecommendationsRequest, check_prototype):
        """
        A method to overwrite the default `predict` method of the Vetiver BaseHandler

        Here follows a documentation that I'd like to see in the API docs.
        """

        ...

Let me know if that answered your question, I'm happy to go digging further to help out!

[krumeto]

Hey @isabelizimm and thank you for coming back as quickly!

Just to clarify, I have two main endpoints - predict and model-card.

def initiate_vetiver_model(vetiver_handler, version = '1.0.0'):
    custom_model = APIHandler(model=vetiver_handler)
    vetiver_model = VetiverModel(custom_model, 
                                model_name = "our name",
                                prototype_data=RecommendationsRequest,
                                versioned=True,
                                metadata={
                                    'known_about_by': ['DS'],
                                    'version': version,
                                    'code_example': "..."
                                })

    return vetiver_model

def model_api():
    # Create an instance of the S2API class
    vetiver_handler = CustomHandler()

    # custom function, 
    vetiver_model = initiate_vetiver_model(vetiver_handler=vetiver_handler)

    def fast_api():
        f_api = FastAPI()
        return f_api
    return vetiver.VetiverAPI(vetiver_model, 
                              check_ptype=True, 
                              app_factory=fast_api)

app = model_api()

# The docstring below is nicely displayed

@app.app.get('/model-card')
async def model_card():
    """
    Metadata about the model from the model as specified in initiate_vetiver_model.
    Contains person of knowledge, version and a code example.
    """
    return app.model.metadata

The model_card()'s docstring is nicely displayed The docstring from the handler_predict() is missing: If there is a way to get it there, it would be quite nice.

Thank you a lot for the description arg. Looks nicely and adds to the documentation. I tried overwriting the describe function and it works, but the displayed docstring gets formatted as a code block and I'd rather use the description directly.

Thank you once again!

--edit-- To futher clarify, I am trying to stick to overwriting the predict method vs. creating a custom endpoint as I'd like to have some consistency in the API's created and most of the rest are standard ML models. Always having predict as your endpoint simplifies docs/communication.

[isabelizimm]

I looked into this more deeply, and the reason why it does not automatically render is due to the fact that vetiver is creating a generic endpoint and then utilizing the handler_predict() function within it, so FastAPI is not aware of the handler_predict() docstring. Some options:

• Vetiver passes in the docstring of handler_predict() as a whole on creation of the /predict endpoint. Everything after the initial description formatted as a code block.
• Vetiver passes in the docstring of handler_predict() as a whole on creation of the /predict endpoint. Parse out just the first lines of the docstring (removing Parameters/Notes/etc). This means there will be no odd code block that includes the rest of the docstring, but might be more brittle depending how long people's descriptions are.
• Vetiver exposes the description parameter, and users can input their choice of description.

I can try out these fixes and open up a PR to give you some flexibility to add in these descriptions!

[krumeto]

Thank you once again, @isabelizimm.

Either of those would be really helpful. Regarding formatting, we have some flexibility as it seems like a lot of the markdown works well even when in docstrings. We will probably be able to shape the docstring in a way that is both helpful when reading the code and helpful as a documentation on the API docs.

For example:

@app.app.get('/model-card')
async def model_card():
    """
    ## Some markdown 

    Metadata about the model from the model as specified in initiate_vetiver_model.
    Contains person of knowledge, version, stored_procedure_used and a code example.

    Another way to markdown
    -----------

        A code block starts:
        from some_package import module
    """
    return app.model.metadata

translates to

[isabelizimm]

I was talking about this issue with @machow and @has2k1 (thanks all! 🎉 ), who realized it actually is indent/dedent issues causing this codeblock in the docs. I've opened a PR that will cause the handler_predict() docstring to appear in the API.

If you'd like to try it out and give feedback, you can install the PR directly:

pip install git+https://github.com/rstudio/vetiver-python.git@fastapi-docs

And once merged, it will be available in an upcoming release. Thank you for raising this!

[krumeto]

Thank you so much, team!