search

googleapis / python-aiplatform

A Python SDK for Vertex AI, a fully managed, end-to-end platform for data science and machine learning.
Apache License 2.0
642 stars 348 forks source link

MatchingEngineIndex create_tree_ah_index fails #2967

Closed jwdavis closed 12 months ago

jwdavis commented 1 year ago

Environment details

Steps to reproduce

V1. Try running Google-published notebook at https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vector-search/intro-textemb-vectorsearch.ipynb - cell with following code fails

create index

my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index( display_name = f"embvs-tutorial-index-{UID}", contents_delta_uri = BUCKET_URI, dimensions = 768, approximate_neighbors_count = 20, distance_measure_type = "DOT_PRODUCT_DISTANCE", )

V2. Try running Google-published notebook at https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vector-search/vector-search-quickstart.ipynb. Cell with following code fails:

create Index

my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index( display_name = f"vs-quickstart-index-{UID}", contents_delta_uri = BUCKET_URI, dimensions = 768, approximate_neighbors_count = 10, )

V3. Try running same code in standalone python on local Mac. Same failure

Stack trace (similar in all examples)

_InactiveRpcError Traceback (most recent call last) File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:65, in _wrap_unary_errors..error_remappedcallable(*args, **kwargs) 64 try: ---> 65 return callable(*args, **kwargs) 66 except grpc.RpcError as exc:

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:1161, in _UnaryUnaryMultiCallable.call(self, request, timeout, metadata, credentials, wait_for_ready, compression) 1155 ( 1156 state, 1157 call, 1158 ) = self._blocking( 1159 request, timeout, metadata, credentials, wait_for_ready, compression 1160 ) -> 1161 return _end_unary_response_blocking(state, call, False, None)

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:1004, in _end_unary_response_blocking(state, call, with_call, deadline) 1003 else: -> 1004 raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INTERNAL details = "Internal error encountered." debug_error_string = "UNKNOWN:Error received from peer ipv4:64.233.191.95:443 {grpc_message:"Internal error encountered.", grpc_status:13, created_time:"2023-11-14T23:25:07.40184234+00:00"}"

The above exception was the direct cause of the following exception:

InternalServerError Traceback (most recent call last) Cell In[16], line 2 1 # create index ----> 2 my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index( 3 display_name = f"embvs-tutorial-index-{UID}", 4 contents_delta_uri = BUCKET_URI, 5 dimensions = 768, 6 approximate_neighbors_count = 20, 7 )

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index.py:523, in MatchingEngineIndex.create_tree_ah_index(cls, display_name, contents_delta_uri, dimensions, approximate_neighbors_count, leaf_node_embedding_count, leaf_nodes_to_search_percent, distance_measure_type, description, labels, project, location, credentials, request_metadata, sync, index_update_method, encryption_spec_key_name) 511 algorithm_config = matching_engine_index_config.TreeAhConfig( 512 leaf_node_embedding_count=leaf_node_embedding_count, 513 leaf_nodes_to_search_percent=leaf_nodes_to_search_percent, 514 ) 516 config = matching_engine_index_config.MatchingEngineIndexConfig( 517 dimensions=dimensions, 518 algorithm_config=algorithm_config, 519 approximate_neighbors_count=approximate_neighbors_count, 520 distance_measure_type=distance_measure_type, 521 ) --> 523 return cls._create( 524 display_name=display_name, 525 contents_delta_uri=contents_delta_uri, 526 config=config, 527 description=description, 528 labels=labels, 529 project=project, 530 location=location, 531 credentials=credentials, 532 request_metadata=request_metadata, 533 sync=sync, 534 index_update_method=index_update_method, 535 encryption_spec_key_name=encryption_spec_key_name, 536 )

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/base.py:817, in optional_sync..optional_run_in_thread..wrapper(*args, *kwargs) 815 if self: 816 VertexAiResourceNounWithFutureManager.wait(self) --> 817 return method(args, **kwargs) 819 # callbacks to call within the Future (in same Thread) 820 internal_callbacks = []

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index.py:209, in MatchingEngineIndex._create(cls, display_name, contents_delta_uri, config, description, labels, project, location, credentials, request_metadata, sync, index_update_method, encryption_spec_key_name) 205 gapic_index.labels = labels 207 api_client = cls._instantiate_client(location=location, credentials=credentials) --> 209 create_lro = api_client.create_index( 210 parent=initializer.global_config.common_location_path( 211 project=project, location=location 212 ), 213 index=gapic_index, 214 metadata=request_metadata, 215 ) 217 _LOGGER.log_create_with_lro(cls, create_lro) 219 created_index = create_lro.result(timeout=None)

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py:592, in IndexServiceClient.create_index(self, request, parent, index, retry, timeout, metadata) 587 metadata = tuple(metadata) + ( 588 gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)), 589 ) 591 # Send the request. --> 592 response = rpc( 593 request, 594 retry=retry, 595 timeout=timeout, 596 metadata=metadata, 597 ) 599 # Wrap the response in an operation future. 600 response = gac_operation.from_gapic( 601 response, 602 self._transport.operations_client, 603 gca_index.Index, 604 metadata_type=index_service.CreateIndexOperationMetadata, 605 )

File /opt/conda/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py:113, in _GapicCallable.call(self, timeout, retry, *args, *kwargs) 110 metadata.extend(self._metadata) 111 kwargs["metadata"] = metadata --> 113 return wrapped_func(args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:67, in _wrap_unary_errors..error_remappedcallable(*args, **kwargs) 65 return callable(*args, **kwargs) 66 except grpc.RpcError as exc: ---> 67 raise exceptions.from_grpc_error(exc) from exc

InternalServerError: 500 Internal error encountered.

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

sasha-gitg commented 12 months ago

This should be fixed with 1.36.4. Please reopen if needed.