List vector ids by prefix

[jhamon]

Problem

Need to implement the new data plane list endpoint.

Solution

• Update generated code. Pull in list endpoint. All changes under pinecone/core are generated from spec files and can be ignored for this review.
• Implement changes for list endpoint in:
  • pinecone/data/index.py main implementation
  • pinecone/grpc/index_grpc.py main implementation
  • tests/integration/data/conftest.py adjustments to test setup, mainly to generate a new namespace to hold vectors for list testing data.
  • tests/integration/data/seed.py to upsert a larger number of vectors, so I would have enough data to page through
  • tests/integration/data/test_list.py
  • tests/integration/data/test_list_errors.py

Open questions

• Do we expect to ever return more data than just {'id': '1'} in the vectors array? For convenience the list() method is currently implemented as a generator function that abstracts the pagination steps and yields a flat list of id values. For a use case where you were going to immediately fetch those ids, this seems ideal. But would be limiting if we ever wanted to return more than just ids here.

Usage

Install the dev client version install "pinecone-client[grpc]"==3.1.0.dev1

REST

from pinecone import Pinecone

pc = Pinecone(api_key='xxx')
index = pc.Index(host='hosturl')

# To iterate over all result pages using a generator function
for ids in index.list(prefix='pref', limit=3, namespace='foo'):
    print(ids) // ['pref1', 'pref2', 'pref3']

# For manual control over pagination
results = index.list_paginated(
    prefix='pref', 
    limit=3, 
    namespace='foo', 
    pagination_token='eyJza2lwX3Bhc3QiOiI5IiwicHJlZml4IjpudWxsfQ=='
)
print(results.namespace)
print([v.id for v in results.vectors])
print(results.pagination.next)
print(results.usage)

GRPC

from pinecone.grpc import PineconeGRPC

pc = PineconeGRPC(api_key='xxx')
index = pc.Index(host='hosturl')

# To iterate over all result pages using a generator function
for ids in index.list(prefix='pref', limit=3, namespace='foo'):
    print(ids) // ['pref1', 'pref2', 'pref3']

# For manual control over pagination
results = index.list_paginated(
    prefix='pref', 
    limit=3, 
    namespace='foo', 
    pagination_token='eyJza2lwX3Bhc3QiOiI5IiwicHJlZml4IjpudWxsfQ=='
)
print(results.namespace)
print([v.id for v in results.vectors])
print(results.pagination.next)
print(results.usage)

Type of Change

• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Testing

Try out the dev version.

pip install "pinecone-client[grpc]"==3.1.0.dev1

[opbenesh]

@jhamon, awesome work. Excited to get that out!

As for your open question: we do not plan to add anything else in the near future, but it theoretically might happen at some point. However if we end up adding just some auxiliary data it might be possible to modify the iterator function to deal with it/strip it away.

@gdj0nes WDYT?

[jhamon]

I removed the full_page check and added in docstrings.