search

googleapis / python-aiplatform

A Python SDK for Vertex AI, a fully managed, end-to-end platform for data science and machine learning.
Apache License 2.0
615 stars 327 forks source link

vertex-aiplatform 1.54 throwing runtime exception: Gapic client context issue . This can occur due to parallelization #4099

Closed lobomacb closed 1 month ago

lobomacb commented 1 month ago

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

If you are still having issues, please be sure to include as much information as possible:

Environment details

Steps to reproduce

Using Vector Text embeddings from SDK google-cloud-aiplatform 1.54, error came suddenly without dependency changes on productive environment.

There was a fix for previous version of this error on https://github.com/googleapis/python-aiplatform/issues/3838 but we are seeing the following error stack trace:

Loading documents to vector store: 0% 0/173665 [00:00<?, ?it/s]
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Loading documents to vector store: 0% 20/173665 [00:01<4:22:25, 11.03it/s]
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Loading documents to vector store: 0% 40/173665 [00:02<3:16:04, 14.76it/s]
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

c0dyhi11 commented 1 month ago

+1

I'm hitting the same issue using LangChain and text-embedding-004

humbroll commented 1 month ago

Got the same issue using from langchain_google_vertexai import ChatVertexAI and from langchain_google_vertexai import VertexAIEmbeddings and textembedding-gecko@003 embedding model.

matthew29tang commented 1 month ago

This should be a non-blocking warning, not an actual exception. Can you confirm if your code successfully runs, or is it a runtime error?

humbroll commented 1 month ago

It was an infinite warning which caused my process is blocked. The log was like below; 

e to parallelization.
2024-07-19 17:21:11,056 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:11,565 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:11,566 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:11,568 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:11,586 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,058 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,089 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,102 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,108 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,519 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,748 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:12,820 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,068 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,150 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,161 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,466 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,645 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:13,655 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:14,007 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:14,365 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:14,523 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:15,022 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:15,042 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:15,574 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:15,638 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,040 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,053 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,058 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,345 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,358 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,490 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,768 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:16,770 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:17,109 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:17,121 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:17,569 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:17,572 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:17,590 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,004 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,006 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,010 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,355 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,379 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,725 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:18,774 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:19,080 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:19,127 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:19,442 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:19,600 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,389 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,427 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,432 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,797 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,824 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,826 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization.
2024-07-19 17:21:20,833 [WARNING] google.cloud.aiplatform.telemetry: Gapic client context issue detected.This can occur due to parallelization.
Gapic client context issue detected.This can occur due to parallelization
matthew29tang commented 1 month ago

While the log output has quite a large volume, it should not block your code (it does not throw an exception).

Anyhow, I've changed the log type to be debug in https://github.com/googleapis/python-aiplatform/commit/0bd4289c5362fc124b843d48a0c5ee4b58db1385, this will be included in the next release.