[Bug]: credential issues when using the Claude model on Vertex AI.

ZeroClover commented 6 days ago

What happened?

Using litellm as a Proxy Server, docker image main latest.

When setting up the Gemini model in Vertex AI, everything was fine, but when I set up the Claude model provided by Vertex AI using the same credentials, the health check run resulted in the following error.

Relevant log output

{
      "vertex_project": "REDACTED",
      "vertex_location": "us-east4",
      "model": "vertex_ai/claude-3-haiku@20240307",
      "vertex_credentials": "{\n  \"type\": \"service_account\",\n  \"project_id\": \"REDACTED\",\n  \"private_key_id\": \"REDACTED\",\n  \"private_key\": \"REDACTED\",\n  \"client_email\": \"litellm@REDACTED.iam.gserviceaccount.com\",\n  \"client_id\": \"REDACTED\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/litellm%40REDACTED.iam.gserviceaccount.com\",\n  \"universe_domain\": \"googleapis.com\"\n}\n",
      "cache": {
        "no-cache": true
      },
      "error": "litellm.APIConnectionError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 380, in acompletion\n    response = await init_response\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py\", line 456, in async_completion\n    message = await vertex_ai_client.messages.create(**data)  # type: ignore\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/resources/messages.py\", line 1474, in create\n    return await self._post(\n           ^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py\", line 1738, in post\n    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py\", line 1441, in request\n    return await self._request(\n           ^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py\", line 1463, in _request\n    await self._prepare_request(request)\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/lib/vertex/_client.py\", line 257, in _prepare_request\n    access_token = await self._ensure_access_token()\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/lib/vertex/_client.py\", line 270, in _ensure_access_token\n    self._credentials, project_id = await asyncify(load_auth)()\n                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_utils/_sync.py\", line 62, in wrapper\n    return await anyio.to_thread.run_sync(partial_f, cancellable=cancellable, limiter=limiter)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anyio/to_thread.py\", line 56, in run_sync\n    return await get_async_backend().run_sync_in_worker_thread(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py\", line 2134, in run_sync_in_worker_thread\n    return await future\n           ^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py\", line 851, in run\n    result = context.run(func, *args)\n             ^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/lib/vertex/_auth.py\", line 21, in load_auth\n    credentials, project_id = google_auth.default(\n                              ^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/google/auth/_default.py\", line 691, in default\n    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)\ngoogle.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.\n stack trace: Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/main.py\", line 380, in acompletion\n    response = await init_response\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py\", line 456, in async_completion\n    message = await vertex_ai_client.messages.create(**data)  # type: ignore\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/resources/messages.py\", line 1474, in create\n    return await self._post(\n           ^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py\", line 1738, in post\n    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py\", line 1441, in request\n    return await self._request"
    }

Twitter / LinkedIn details

No response

Manouchehri commented 3 days ago

imo duplicate of #3957.

ZeroClover commented 3 days ago

This is not just a health check issue, it happen when actually calling the API.

ishaan-jaff commented 2 days ago

@ZeroClover how do you set your vertex credentials on LiteLLM? Is this the service account.json ?

ZeroClover commented 2 days ago

@ishaan-jaff Yes, I'm using the service account key json.

smithla02 commented 2 days ago

I had a coworker mention that he thought this URL in vertex_httpx.py only supports models published by google, and not anthropic. We were also having trouble getting the Claude 3.5 Sonnet model on Vertex AI to work.

I am not sure if that's the source of the issue?

krrishdholakia commented 1 day ago

hey @smithla02 anthropic calls go here - https://github.com/BerriAI/litellm/blob/2452753e084e8134c0c484b32c63fb5f2950c5ba/litellm/llms/vertex_ai_anthropic.py#L206

krrishdholakia commented 1 day ago

is this on latest @smithla02, we had users report this, but it's since been fixed

BerriAI / litellm