Add configuration specification for Pinecode index

[friofry]

I've created a pinecone index with dimension 3072, and I get this error when running python src/index.py $GITHUB_REPO_NAME --pinecone_index_name=$PINECONE_INDEX_NAME.

I think it would be helpful to mention in the docs that the index should be created with dimension 1536.

INFO:httpx:HTTP Request: GET https://api.openai.com/v1/files/file-FEcByztU5sHBQ3zfezDO8ETE/content "HTTP/1.1 200 OK"
INFO:root:Job batch_D1aOpYvQzZH7OGtfSLhimhBr generated 2000 embeddings.
Traceback (most recent call last):
  File "~/repo2vec/src/index.py", line 107, in <module>
    main()
  File "~/repo2vec/src/index.py", line 102, in main
    vector_store.upsert(embedder.download_embeddings())
  File "~/repo2vec/src/vector_store.py", line 27, in upsert
    self.upsert_batch(batch)
  File "~/repo2vec/src/vector_store.py", line 54, in upsert_batch
    self.index.upsert(vectors=pinecone_vectors, namespace=self.namespace)
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/utils/error_handling.py", line 11, in inner_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/data/index.py", line 175, in upsert
    return self._upsert_batch(vectors, namespace, _check_type, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/data/index.py", line 204, in _upsert_batch
    return self._vector_api.upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 761, in __call__
    return self.callable(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/data/api/data_plane_api.py", line 811, in __upsert
    return self.call_with_http_info(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 819, in call_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 380, in call_api
    return self.__call_api(
           ^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 187, in __call_api
    raise e
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 175, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/api_client.py", line 460, in request
    return self.rest_client.POST(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/rest.py", line 345, in POST
    return self.request(
           ^^^^^^^^^^^^^
  File "~/repo2vec/.venv/lib/python3.12/site-packages/pinecone/core/openapi/shared/rest.py", line 279, in request
    raise PineconeApiException(http_resp=r)
pinecone.core.openapi.shared.exceptions.PineconeApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'Content-Length': '104', 'x-pinecone-request-latency-ms': '744', 'x-pinecone-request-id': '8710845252429336766', 'date': 'Sat, 31 Aug 2024 16:45:50 GMT', 'x-envoy-upstream-service-time': '153', 'server': 'envoy', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: {"code":3,"message":"Vector dimension 1536 does not match the dimension of the index 3072","details":[]}

[iuliaturc]

Thanks for flagging this! You can now define custom OpenAI embedding sizes. For instance:

Change default embedding model:

python src/index.py iuliaturc/detextify --embedder_type=openai --embedding_model=text-embedding-3-large --vector_store_type=pinecone --index_name=test-3072

Change default embedding model AND reduce the default embedding size:

python src/index.py iuliaturc/detextify --embedder_type=openai --embedding_model=text-embedding-3-large --embedding_size=256 --vector_store_type=pinecone --index_name=test-256