[Bug]: litellm not working with vertex ai when using service account json file

[vaghelarahul94]

What happened?

When using litellm SDK version 1.48.7 like this:

from litellm import completion
import json

## GET CREDENTIALS
file_path = 'PATH_TO_JSON'

# Load the JSON file
with open(file_path, 'r') as file:
    vertex_credentials = json.load(file)

# Convert to JSON string
vertex_credentials_json = json.dumps(vertex_credentials)

response = completion(
  model="vertex_ai/gemini-pro",
  messages=[{"content": "You are a good bot.","role": "system"}, {"content": "tell me poem on pasta","role": "user"}],
  vertex_credentials=vertex_credentials_json,
  vertex_project="my_project_id",
  vertex_location="us-central1"
)

We are seeing error with stack trace:

Traceback (most recent call last):
  File "/home/appuser/.local/lib/python3.11/site-packages/litellm/main.py", line 2280, in completion
    model_response = vertex_chat_completion.completion(  # type: ignore
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1208, in completion
    _auth_header, vertex_project = self._ensure_access_token(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/vertex_llm_base.py", line 137, in _ensure_access_token
    self._credentials, cred_project_id = self.load_auth(
                                         ^^^^^^^^^^^^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/vertex_llm_base.py", line 79, in load_auth
    creds = identity_pool.Credentials.from_info(json_obj)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/google/auth/identity_pool.py", line 425, in from_info
    return super(Credentials, cls).from_info(info, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/google/auth/external_account.py", line 591, in from_info
    return cls(
           ^^^^
  File "/home/appuser/.local/lib/python3.11/site-packages/google/auth/identity_pool.py", line 273, in __init__
    raise exceptions.MalformedError(
google.auth.exceptions.MalformedError: Invalid Identity Pool credential_source field 'environment_id'

Whereas when I use vertex ai SDK itself like this with my same service account credentials file like this:

import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="PATH_TO_JSON"
os.environ["VERTEXAI_LOCATION"]="us-central1"
os.environ["VERTEXAI_PROJECT"]="my_project_id""

import vertexai
from vertexai.generative_models import GenerativeModel

vertexai.init(project="my_project_id"", location="us-central1")

model = GenerativeModel("gemini-pro")

response = model.generate_content(
    "tell me poem on pasta"
)

print(response.text)

Then it works fine.

My service account file looks like this:

{
  "type": "external_account",
  "audience": "....",
  "subject_token_type": "....",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/.....iam.gserviceaccount.com:generateAccessToken",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "environment_id": "aws1",
    "region_url": "....",
    "url": "....",
    "regional_cred_verification_url": "...."
  }
}

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

google.auth.exceptions.MalformedError: Invalid Identity Pool credential_source field 'environment_id'

this error is coming from the google sdk not litellm

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="PATH_TO_JSON"

Try doing using the env var for litellm, and see if that works instead @vaghelarahul94

[vaghelarahul94]

Hi @krrishdholakia Thanks for your response! I appreciate it.

I updated the code to use the os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="PATH_TO_JSON" environment variable. Additionally, I commented out a few other sections.

response = completion(
  model="vertex_ai/gemini-pro",
  messages=[{"content": "You are a good bot.","role": "system"}, {"content": "tell me poem on pasta","role": "user"}],
  #vertex_credentials=vertex_credentials_json,
  #vertex_project="my_project_id",
  #vertex_location="us-central1"
)

I am now seeing that litellm SDK is asking for this role serviceUsageConsumer ? Whereas vertex ai sdk didn't ask for it. Why when using via litellm sdk this is happening?

litellm.exceptions.BadRequestError: litellm.BadRequestError: VertexAIException BadRequestError - ('Unable to acquire impersonated credentials', '{\n "error": {\n "code": 403,\n "message": "Caller does not have required permission to use project my_project_id. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=my_project_id and then retry. Propagation of the new permission may take a few minutes.",\n "status": "PERMISSION_DENIED",\n "details": [\n {\n "@type": "type.googleapis.com/google.rpc.Help",\n "links": [\n {\n "description": "Google developer console IAM admin",\n "url": "https://console.developers.google.com/iam-admin/iam/project?project=my_project_id"\n }\n ]\n },\n {\n "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n "reason": "USER_PROJECT_DENIED",\n "domain": "googleapis.com",\n "metadata": {\n "service": "iamcredentials.googleapis.com",\n "consumer": "projects/my_project_id"\n }\n }\n ]\n }\n}\n')