Multimodal embedding failed with the following error: 401 Deadline UNAUTHENTICATED

[jackyliang]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

• Search the issues already opened: https://github.com/googleapis/python-aiplatform/issues
• Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS type and version: Mac OS X M2 Sonoma 14.4.1
• Python version: Python 3.12.2
• pip version: pip 23.3.1
• google-cloud-aiplatform version: 1.49.0

Steps to reproduce

1. Try the Python example code on Vertex AI's Multimodal Embedding for video (link: https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings#video-modes)
2. Run it
3. Get the following error: Video embedding failed with the following error: Deadline

Code example

This is exactly the code shared in the Vertex AI Multimodal Embedding Model documentation, with the only addition being my variables and the link to the Google Cloud Storage. I have tested both with a local video file AND a gcsUri path

    import vertexai

    from vertexai.vision_models import MultiModalEmbeddingModel, Video

    project_id = 'multimodal-project'
    location = 'us-central1'

    vertexai.init(project=project_id, location=location)

    # Document metadata
    video_path = 'gs://multi-assets/videos/dog_jumping_short.mp4' # small 1 MB 7 second video file
    description = 'dogs jumping'

    model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")

    video = Video.load_from_file(video_path)

    embeddings = model.get_embeddings(
        video=video
    )

# Video Embeddings are segmented based on the video_segment_config.
print("Video Embeddings:")
for video_embedding in embeddings.video_embeddings:
    print(
        f"Video Segment: {video_embedding.start_offset_sec} - {video_embedding.end_offset_sec}"
    )
    print(f"Embedding: {video_embedding.embedding}")

print(f"Text Embedding: {embeddings.text_embedding}")

Stack trace

Traceback (most recent call last):
  File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Video embedding failed with the following error: Deadline"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.81.234:443 {grpc_message:"Video embedding failed with the following error: Deadline", grpc_status:16, created_time:"2024-05-23T15:53:15.429704-04:00"}"

Again, I have tried not only the Python sample code, but the cURL commands and Cloud Shell (which should fix all authentication issues, because you need to authenticate prior to use), but they all return the same error.

I have no idea how to fix this issue, and I've tried to isolate as many things as possible.

Workaround

If you are a dev using this, you can workaround the native video embedding not working by using the base64 string method. Convert a video (you will be limited by the video length / size to around 2-3 minute video) to base64 string, then load into the bytesBase64Encoded variable instead of gcsUri

[jackyliang]

@holtskinner any chance you may know the reason why?

[jackyliang]

I am trying two other things, using a Public Bucket, and double checking gcs bucket permissions

1. Public bucket

I have the following public URL video (gs://test-public-bucket-123/dog_jumping_short.mp4), and I still get the same error unfortunately.

2. Using gsutil ls to check whether after-authentication, I can find my private bucket file

--

Both continue to fail with the same error :/

[jaycee-li]

Hi @jackyliang , are you still getting the same error? I tried your code and it worked on my side.

The error message you're seeing suggests a problem with authentication when calling the API, rather than a permission issue with the GCS bucket. To isolate the cause, could you test get_embeddings() using text input only and see if it works?

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
embeddings = model.get_embeddings(
    contextual_text="hello world"
)