search

swarmauri / swarmauri-sdk

https://swarmauri.com
Apache License 2.0
9 stars 21 forks source link

[API Change Request - Community]: llms/ShuttleAIModel.py and llms/ShuttleAIToolModel.py #177

Open cobycloud opened 2 weeks ago

cobycloud commented 2 weeks ago

Affected component

llms/ShuttleAIToolModel.py

Motivation

Our testing indicates changes in the ShuttleAIModel, which have surfaced JSON-related errors:

-- FAILED tests/llms/ShuttleAIModel_test.py::test_no_system_context - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) FAILED tests/llms/ShuttleAIModel_test.py::test_nonpreamble_system_context - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) FAILED tests/llms/ShuttleAIModel_test.py::test_preamble_system_context - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) FAILED tests/llms/ShuttleAIModel_test.py::test_multiple_system_contexts - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) FAILED tests/llms/ShuttleAIToolModel_test.py::test_agent_exec - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Potential Solutions

No response

Additional Context (optional)

No response

Priority

Low

cobycloud commented 2 weeks ago

tests/llms/ShuttleAIModel_test.py::test_no_system_context -------------------------------- live log call ---------------------------------

in this test: https://github.com/swarmauri/swarmauri-sdk/actions/runs/10511376355/job/29151216114

Payload being sent: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Hello', 'role': 'user'}], 'max_tokens': 256, 'temperature': 0.7, 'top_p': 1}
Response received: <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>

<title>api.shuttleai.app | 524: A timeout occurred</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" />

</head>
<body>
<div id="cf-wrapper">
    <div id="cf-error-details" class="p-0">
        <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">
            <h1 class="inline-block sm:block sm:mb-2 font-light text-60 lg:text-4xl text-black-dark leading-tight mr-2">
              <span class="inline-block">A timeout occurred</span>
              <span class="code-label">Error code 524</span>
            </h1>
            <div>
               Visit <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=api.shuttleai.app" target="_blank" rel="noopener noreferrer">cloudflare.com</a> for more information.
            </div>
            <div class="mt-3">2024-08-23 06:43:45 UTC</div>
        </header>
        <div class="my-8 bg-gradient-gray">
            <div class="w-240 lg:w-full mx-auto">
                <div class="clearfix md:px-8">

<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-browser block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">You</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Browser

  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=api.shuttleai.app" target="_blank" rel="noopener noreferrer">
    <span class="cf-icon-cloud block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
    </a>
  </div>
  <span class="md:block w-full truncate">Ashburn</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=api.shuttleai.app" target="_blank" rel="noopener noreferrer">
    Cloudflare
    </a>
  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-server block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">api.shuttleai.app</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Host

  </h3>
  <span class="leading-1.3 text-2xl text-red-error">Error</span>
</div>

                </div>
            </div>
        </div>

        <div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
            <div class="clearfix">
                <div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What happened?</h2>
                    <p>The origin web server timed out responding to this request.</p>
                </div>
                <div class="w-1/2 md:w-full float-left leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What can I do?</h2>
                          <h3 class="text-15 font-semibold mb-2">If you're a visitor of this website:</h3>
      <p class="mb-6">Please try again in a few minutes.</p>

      <h3 class="text-15 font-semibold mb-2">If you're the owner of this website:</h3>
      <p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a rel="noopener noreferrer" href="https://support.cloudflare.com/hc/en-us/articles/200171926-Error-524">Additional troubleshooting information here.</a></p>
                </div>
            </div>
        </div>

        <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">8b791edeaf3[680](https://github.com/swarmauri/swarmauri-sdk/actions/runs/10511376355/job/29151216114#step:9:681)06</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">74.249.14.208</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=api.shuttleai.app" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->

    </div>
</div>
</body>
</html>

FAILED                                                                   [ 57%]
cobycloud commented 2 weeks ago

in workflow #485 https://github.com/swarmauri/swarmauri-sdk/actions/runs/10511110197/job/29121314771

tests/llms/ShuttleAIModel_test.py::test_no_system_context 
-------------------------------- live log call ---------------------------------
Payload being sent: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Hello', 'role': 'user'}], 'max_tokens': 256, 'temperature': 0.7, 'top_p': 1}
Response received: {
  "id": "chatcmpl-d33722f38e114da6bed0d272d97e7029",
  "object": "chat.completion",
  "created": 1724340874,
  "model": "shuttle-2.5",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 9,
    "total_charged": 9.8e-6
  }
}
PASSED
cobycloud commented 2 weeks ago

from run #487

https://github.com/swarmauri/swarmauri-sdk/actions/runs/10522871149

tests/llms/ShuttleAIModel_test.py::test_no_system_context -------------------------------- live log call --------------------------------- Payload being sent: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Hello', 'role': 'user'}], 'max_tokens': 256, 'temperature': 0.7, 'top_p': 1} Response received: { "id": "chatcmpl-ad39c53546744b8284ecf7d1cbf6d2f1", "object": "chat.completion", "created": 1724403484, "model": "shuttle-2.5", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! How can I assist you today?" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 7, "completion_tokens": 9, "total_charged": 9.8e-6 } } PASSED

cobycloud commented 2 weeks ago

from run #487

https://github.com/swarmauri/swarmauri-sdk/actions/runs/10522871149/job/29156418314 tests/llms/ShuttleAIToolModel_test.py::test_agent_exec -------------------------------- live log call --------------------------------- payload: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Add 512+671', 'role': 'user'}], 'max_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'AdditionTool', 'description': 'This tool has two numbers together', 'parameters': {'type': 'object', 'properties': {'x': {'type': 'integer', 'description': 'The left operand'}, 'y': {'type': 'integer', 'description': 'The right operand'}}, 'required': ['x', 'y']}}}]} agent response {'id': 'chatcmpl-246a4a83762d404480a7cab9e4f35bcb', 'object': 'chat.completion', 'created': 1724403487, 'model': 'shuttle-2.5', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'call_c44bc87b68c34aa08bd1db1b87107258', 'type': 'function', 'function': {'name': 'AdditionTool', 'arguments': '{"x":512,"y":671}'}}]}, 'finish_reason': 'tool_calls'}], 'usage': {'prompt_tokens': 39, 'completion_tokens': 38, 'total_charged': 4.6099999999999996e-05}} conversation: [HumanMessage(name=None, id='5baf2bea-f331-40ce-b9a6-f7f998ffc430', members=[], owner=None, host=None, resource='Message', version='0.1.0', type='HumanMessage', content='Add 512+671', role='user'), FunctionMessage(name='AdditionTool', id='57bad6fa-7a19-40a8-9ff0-45e61a5a3659', members=[], owner=None, host=None, resource='Message', version='0.1.0', type='FunctionMessage', content='1183', role='tool', tool_call_id='call_c44bc87b68c34aa08bd1db1b87107258', usage=None)] payload: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Add 512+671', 'role': 'user'}, {'name': 'AdditionTool', 'content': '1183', 'role': 'tool', 'tool_call_id': 'call_c44bc87b68c34aa08bd1db1b87107258'}], 'max_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'AdditionTool', 'description': 'This tool has two numbers together', 'parameters': {'type': 'object', 'properties': {'x': {'type': 'integer', 'description': 'The left operand'}, 'y': {'type': 'integer', 'description': 'The right operand'}}, 'required': ['x', 'y']}}}]} agent response {'status': 400, 'error': {'message': 'We could not parse the JSON body of your request.', 'type': 'invalid_request_error', 'code': None, 'param': None, 'hint': "This likely means you aren't using your HTTP library correctly. The ShuttleAI API expects a JSON payload, but what was sent was not valid JSON. If you have trouble figuring out how to fix this, please contact us at https://discord.gg/shuttleai"}} FAILED

cobycloud commented 2 weeks ago

from #478 https://github.com/swarmauri/swarmauri-sdk/actions/runs/10507449907/job/29110233302

tests/llms/ShuttleAIToolModel_test.py::test_agent_exec -------------------------------- live log call --------------------------------- payload: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Add 512+671', 'role': 'user'}], 'max_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'AdditionTool', 'description': 'This tool has two numbers together', 'parameters': {'type': 'object', 'properties': {'x': {'type': 'integer', 'description': 'The left operand'}, 'y': {'type': 'integer', 'description': 'The right operand'}}, 'required': ['x', 'y']}}}], 'internet': True} agent response {'id': 'chatcmpl-a50dd1ce801946bd80dcb01afae0bc9a', 'object': 'chat.completion', 'created': 1724328809, 'model': 'shuttle-2.5', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': 'Searching the internet for more information... Results outputted in next user message.'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 39, 'completion_tokens': 15, 'total_charged': 2.9999999999999997e-05}} conversation: [HumanMessage(name=None, id='b5bdaefb-3d40-4c42-94c0-e30103f14232', members=[], owner=None, host=None, resource='Message', version='0.1.0', type='HumanMessage', content='Add 512+671', role='user')] payload: {'model': 'shuttle-2-turbo', 'messages': [{'content': 'Add 512+671', 'role': 'user'}], 'max_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'AdditionTool', 'description': 'This tool has two numbers together', 'parameters': {'type': 'object', 'properties': {'x': {'type': 'integer', 'description': 'The left operand'}, 'y': {'type': 'integer', 'description': 'The right operand'}}, 'required': ['x', 'y']}}}], 'internet': True} agent response {'id': 'chatcmpl-e91a8520fa7d4359a7486ec3246c7395', 'object': 'chat.completion', 'created': 1724328812, 'model': 'shuttle-2.5', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': 'None of the search results provided contain information about the sum of 512+671. The search results appear to be about roofing companies and phone numbers in the Austin, Texas area. I apologize for any confusion. Let me know if you have any other questions!'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 39, 'completion_tokens': 56, 'total_charged': 5.87e-05}} conversation: [HumanMessage(name=None, id='b5bdaefb-3d40-4c42-94c0-e30103f14232', members=[], owner=None, host=None, resource='Message', version='0.1.0', type='HumanMessage', content='Add 512+671', role='user'), AgentMessage(name=None, id='10942b19-2554-4aa7-b781-9a79200df3ac', members=[], owner=None, host=None, resource='Message', version='0.1.0', type='AgentMessage', content='None of the search results provided contain information about the sum of 512+671. The search results appear to be about roofing companies and phone numbers in the Austin, Texas area. I apologize for any confusion. Let me know if you have any other questions!', role='assistant', usage=None)] None of the search results provided contain information about the sum of 512+671. The search results appear to be about roofing companies and phone numbers in the Austin, Texas area. I apologize for any confusion. Let me know if you have any other questions! PASSED