search

twelvelabs-io / twelvelabs-python

Official TwelveLabs SDK for Python
https://pypi.org/project/twelvelabs
Apache License 2.0
27 stars 4 forks source link
python twelvelabs

readme

TwelveLabs Python SDK

PyPI version

This SDK provides a convenient way to interact with the Twelve Labs Video Understanding Platform from an application written in the Python language. The SDK equips you with a set of intuitive classes and methods that streamline the process of interacting with the platform, minimizing the need for boilerplate code.

Prerequisites

Ensure that the following prerequisites are met before using the SDK:

Install the twelvelabs package:

pip install twelvelabs

Initialize the SDK

  1. Import the SDK into your application:

    from twelvelabs import TwelveLabs

  2. Instantiate the SDK client with your API key. This example code assumes that your API key is stored in an environment variable named TL_API_KEY:

    client = TwelveLabs(api_key=os.getenv('TL_API_KEY'))

Use the SDK

To get started with the SDK, follow these basic steps:

  1. Create an index.
  2. Upload videos.
  3. Perform downstream tasks, such as searching or generating text from video.

Create an index

To create an index, use the example code below, replacing "" with the desired name for your index:

from twelvelabs import APIStatusError

index_obj = None
try:
    index_obj = client.index.create(
        name = "<YOUR_INDEX_NAME>",
        engines =[
            {
                "name": "marengo2.6",
                "options": ["visual", "conversation", "text_in_video"],
            },
            {
                "name": "pegasus1.1",
                "options": ["visual", "conversation"],
            },
        ],
    )
    print(index_obj)
except APIStatusError as e:
    print('API Status Error, 4xx or 5xx')
    print(e)
except Exception as e:
    print(e)

Note the following about this example:

The output should look similar to the following:

Index(id='65b1b926560f741da96836d7', created_at='2024-01-25T01:28:06.061Z', updated_at='2024-01-25T01:28:06.061Z', name='test-index-to-researchers1', engines=[Engine(name='marengo2.6', options=['visual', 'conversation', 'text_in_video'], addons=None), Engine(name='pegasus1.1', options=['visual', 'conversation'], addons=None)], video_count=0, total_duration=0.0, expires_at='2024-04-24T01:28:06.061Z')

Note that the API returns, among other information, a field named id, representing the unique identifier of your new index.

For a description of each field in the request and response, see the Create an index page.

Upload videos

Before you upload a video to the platform, ensure that it meets the following requirements:

To upload videos, use the example code below, replacing the following:

from glob import glob
from twelvelabs.models.task import Task

video_files = glob("<YOUR_VIDEO_PATH>") # Example: "/videos/*.mp4
for video_file in video_files:
  print(f"Uploading {video_file}")
  task = client.task.create(index_id="<YOUR_INDEX_ID>", file=video_file, language="en")
  print(f"Task id={task.id}")

  # (Optional) Monitor the video indexing process
  # Utility function to print the status of a video indexing task
  def on_task_update(task: Task):
          print(f"  Status={task.status}")
  task.wait_for_done(callback=on_task_update)
  if task.status != "ready":
      raise RuntimeError(f"Indexing failed with status {task.status}")
  print(f"Uploaded {video_file}. The unique identifer of your video is {task.video_id}.")

Note that once a video has been successfully uploaded and indexed, the response will contain a field named video_id, representing the unique identifier of your video.

For a description of each field in the request and response, see the Create a video indexing task page.

Perform downstream tasks

The sections below show how you can perform the most common downstream tasks. See our documentation for a complete list of all the features the Twelve Labs Understanding Platform provides.

Search

To search for relevant video content, you can use either text or images as queries:

Search using text queries

To perform a search request using text queries, use the example code below, replacing the following:

search_results = client.search.query(
  index_id="<YOUR_INDEX_ID>",
  query_text="<YOUR_QUERY>",
  options=["<YOUR_SEARCH_OPTIONS>"]
)

# Utility function to print a specific page
def print_page(page):
  for clip in page:
    print(
        f" video_id={clip.video_id} score={clip.score} start={clip.start} end={clip.end} confidence={clip.confidence}"
    )

print_page(search_results.data)

while True:
    try:
        print_page(next(search_results))
    except StopIteration:
        break

The results are returned one page at a time, with a default limit of 10 results on each page. The next method returns the next page of results. When you've reached the end of the dataset, a StopIteration exception is raised.

 video_id=65ca2bce48db9fa780cb3fa4 score=84.9 start=104.9375 end=111.90625 confidence=high
 video_id=65ca2bce48db9fa780cb3fa4 score=84.82 start=160.46875 end=172.75 confidence=high
 video_id=65ca2bce48db9fa780cb3fa4 score=84.77 start=55.375 end=72.46875 confidence=high

Note that the response contains, among other information, the following fields:

For a description of each field in the request and response, see the Make any-to-video search requests page.

Search using image queries

You can provide images as local files or publicly accessible URLs. Use the query_media_file parameter for local image files and the query_media_url parameter for publicly accessible URLs.

To perform a search request using image queries, use the example code below, replacing the following:

search_results = client.search.query(
    index_id="<YOUR_INDEX_ID>",
    query_media_type="image",
    query_media_file="<YOUR_FILE_PATH>", # Use query_media_url instead to provide a file from a publicly accessible URL.
    options=["<YOUR_SEARCH_OPTIONS>"]
)

The response is similar to that received when using text queries.

Generate text from video

The Twelve Labs Video Understanding Platform offers three distinct endpoints tailored to meet various requirements. Each endpoint has been designed with specific levels of flexibility and customization to accommodate different needs.

Note the following about using these endpoints:

Topics, titles, and hashtags

To generate topics, titles, and hashtags, use the example code below, replacing the following:

res = client.generate.gist("<YOUR_VIDEO_ID>", types=["<TYPES>"])
print(f"Title = {res.title}\nTopics = {res.topics}\nHashtags = {res.hashtags}")

For a description of each field in the request and response, see the Titles, topics, or hashtags page.

Summaries, chapters, and highlights

To generate summaries, chapters, and highlights, use the example code below, replacing the following:

res = client.generate.summarize("<YOUR_VIDEO_ID>", type="<TYPE>", prompt="<YOUR_PROMPT>")
print(f"{res.summary}")

For a description of each field in the request and response, see the Summaries, chapters, or highlights page.

Open-ended texts

To generate open-ended texts, use the example code below, replacing the following:

res = client.generate.text(video_id="<YOUR_VIDEO_ID>", prompt="<YOUR_PROMPT>")
print(f"{res.data}")

Error Handling

The SDK includes a set of exceptions that are mapped to specific HTTP status codes, as shown in the table below:

Exception HTTP Status Code
BadRequestError 400
AuthenticationError 401
PermissionDeniedError 403
NotFoundError 404
ConflictError 409
UnprocessableEntityError 422
RateLimitError 429
InternalServerError 5xx

The following example shows how you can handle specific HTTP errors in your application:

import os
from twelvelabs import TwelveLabs

client = TwelveLabs(os.getenv("TWELVELABS_API_KEY"))
try:
    engines = client.engine.list()
    print(engines)
except twelvelabs.APIConnectionError as e:
    print("Cannot connect to API server")
except twelvelabs.BadRequestError as e:
    print("Bad request.")
except twelvelabs.APIStatusError as e:
    print(f"Status code {e.status_code} received")
    print(e.response)

License

We use the Developer Certificate of Origin (DCO) in lieu of a Contributor License Agreement for all contributions to Twelve Labs' open-source projects. We request that contributors agree to the terms of the DCO and indicate that agreement by signing all commits made to Twelve Labs' projects by adding a line with your name and email address to every Git commit message contributed, as shown in the example below:

Signed-off-by: Jane Doe <jane.doe@example.com>

You can sign your commit automatically with Git by using git commit -s if you have your user.name and user.email set as part of your Git configuration. We ask that you use your real name (please, no anonymous contributions or pseudonyms). By signing your commitment, you are certifying that you have the right have the right to submit it under the open-source license used by that particular project. You must use your real name (no pseudonyms or anonymous contributions are allowed.) We use the Probot DCO GitHub app to check for DCO signoffs of every commit. If you forget to sign your commits, the DCO bot will remind you and give you detailed instructions for how to amend your commits to add a signature.