[Bug]: Aws Sagemaker hosted models does not work

[khanzzirfan]

What happened?

env setup

os.environ["AWS_ACCESS_KEY_ID"] = "xxxx" os.environ["AWS_SECRET_ACCESS_KEY"] = "xxxxxx" os.environ["AWS_REGION_NAME"] = "ap-southeast-2"

response = completion( model="sagemaker/jumpstart-dft-meta-textgeneration-l-20240xxxx6-xxxxxx", messages=[{ "content": "write me a function to print hello world?","role": "user"}], temperature=0.2, max_tokens=800 )

never worked. I see only errors.

Relevant log output

Attaching to litellm-codellama-server-litellm-1
litellm-codellama-server-litellm-1  | ERROR:main:Exception on /chat/completions [POST]
litellm-codellama-server-litellm-1  | Traceback (most recent call last):
litellm-codellama-server-litellm-1  |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1473, in wsgi_app
litellm-codellama-server-litellm-1  |     response = self.full_dispatch_request()
litellm-codellama-server-litellm-1  |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 883, in full_dispatch_request
litellm-codellama-server-litellm-1  |     return self.finalize_request(rv)
litellm-codellama-server-litellm-1  |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 902, in finalize_request
litellm-codellama-server-litellm-1  |     response = self.make_response(rv)
litellm-codellama-server-litellm-1  |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1174, in make_response
litellm-codellama-server-litellm-1  |     raise TypeError(
litellm-codellama-server-litellm-1  | TypeError: The view function for 'api_completion' did not return a valid response. The function either returned None or ended without a return statement.

Twitter / LinkedIn details

No response

[krrishdholakia]

None of this error is from litellm @khanzzirfan the error stack is from flask's app.py file. If you have a litellm related stacktrace please reopen the issue with that.

Attaching to litellm-codellama-server-litellm-1 litellm-codellama-server-litellm-1 | ERROR:main:Exception on /chat/completions [POST] litellm-codellama-server-litellm-1 | Traceback (most recent call last): litellm-codellama-server-litellm-1 | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1473, in wsgi_app litellm-codellama-server-litellm-1 | response = self.full_dispatch_request() litellm-codellama-server-litellm-1 | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 883, in full_dispatch_request litellm-codellama-server-litellm-1 | return self.finalize_request(rv) litellm-codellama-server-litellm-1 | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 902, in finalize_request litellm-codellama-server-litellm-1 | response = self.make_response(rv) litellm-codellama-server-litellm-1 | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1174, in make_response litellm-codellama-server-litellm-1 | raise TypeError( litellm-codellama-server-litellm-1 | TypeError: The view function for 'api_completion' did not return a valid response. The function either returned None or ended without a return statement.

[khanzzirfan]

Why have you closed the task. Instead you could have give a try running the AWS sagemaker models test to check if in reality the litellm works on your side. As Authors / Maintainers you would have atleast checked it for me instead right? In fact I have set the verbose flag = True. yet, I did not get any output or errors. this is all what I got.

Just FYI: If I use postman curl request directly to sagemaker invoke-endpoint, the model works perfectly fine and returns responses.

[khanzzirfan]

@krrishdholakia @ishaan-jaff

[krrishdholakia]

@khanzzirfan

we test for sagemaker already - https://github.com/BerriAI/litellm/blob/3bf6589fab8da3d752d71693593834cc59c2da5c/litellm/tests/test_sagemaker.py

Your issue is closed as the stacktrace doesn't contain litellm, which means it's not a litellm error.

Feel free to reopen / create a new issue if you see litellm in your stacktrace. currently it seems like a flask issue.

[khanzzirfan]

hey @krrishdholakia , you are unit testing the work. not the integration mate. If i'm not wrong, you only got Mock magic test cases. Try working with real integration to prove it works please. https://github.com/BerriAI/litellm/blob/3bf6589fab8da3d752d71693593834cc59c2da5c/litellm/tests/test_sagemaker.py#L245

Another thing is, I dont have access to re-open the ticket. cc @ishaan-jaff

[krrishdholakia]

read this @khanzzirfan https://github.com/BerriAI/litellm/blob/3bf6589fab8da3d752d71693593834cc59c2da5c/litellm/tests/test_sagemaker.py#L51

[khanzzirfan]

Thank you so much for the reply @krrishdholakia . I'll try to work my way to see why exactly meta models doesn't work.