Closed flaky-bot[bot] closed 2 years ago
commit: 1b45e8ed2381f50ed16bcc36415f2f9e1278b664 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 7c48837534a86651b1f803dadf930a5160b1c29c buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 13976517f09f19610be1875c0f1686a852ed2864 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 558844ad256c42fa54d34f50d91b6ca0387968ac buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 63d1e3d4ad9d46f8dacfcd159f8d784a8fcd4483 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 83e8ee921080534bab2cbd07289f6f3e7b8e2b8e buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
I've filed a request to increase the quota for project python-docs-samples-tests-py37. I received a reply
This message is to confirm that we've received your quota request for
project 'python-docs-samples-tests-py37'. Quota increase requests typically
take two business days to process. If this quota request is very urgent,
please respond to this email so that our support agents can have full
context when processing your quota increase request.
commit: 3f81e3672444635357bc218c1775768ab297e717 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: fe8dd4ce6953f40afa7e6b5b797b22e7f4daf546 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
@parthea Has that quota increase been granted?
Not yet
commit: 3b6d797140afa6ca900a1e38a554795bfe800368 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 029e84c9e54cce4995acfa167e39d8b565f664d8 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id deployed_model_id = deploy_model.deploy_model( File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: 6c4c8de274496300e3285168a012f8b6203b5122 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
I received an email that says quota has been adjusted but the error still appears
Your quota request for python-docs-samples-tests-py37 has been approved and your project quota
has been adjusted accordingly. To verify the quota change, please navigate to
https://console.cloud.google.com/, select your project, select Google Cloud
AutoML under APIs, and then click on "Quotas" tab on the top of the page.
________________________ ERROR at setup of test_predict ________________________
[gw2] linux -- Python 3.7.12 /workspace/people-and-planet-ai/image-classification/.nox/py-3-7/bin/python
Traceback (most recent call last):
File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
deployed_model = response.result()
File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
raise self._exception
google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
It turns out that the project with a quota issue is python-docs-samples-tests
not python-docs-samples-tests-py37
. I saw python-docs-samples-tests-py37
in the build log so I expect it to use that project.
I've submitted a new request for the correct project.
commit: 3b0d5473f9c1bdd89df3ccec9f369d1b2ea28635 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
commit: f7df2bbb6820324c115cc6e20c698366e2b62ae0 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
I’m going to close this issue and see if it reappears. I already submitted a request to increase the quota.
Looks like this issue is flaky. :worried:
I'm going to leave this open and stop commenting.
A human should fix and close this.
commit: c35e0a66524e3246c2d5366e2e0ddb48d6a555cd buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
Still in progress:
This is an update for your quota increase request. Our Systems engineers are now working on your case. We will keep you informed as to any and all significant updates as soon as we receive them from the concerned teams. Please expect an update from us in the next 5 days.
I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: 1062b16ff66a587bd961fb026742c3bb6502e9a0 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model deployed_model = response.result() File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result raise self._exception google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: AutoMLImageClassificationDeployedModelNodes
The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: cf1643c8e2818c539d0c204405018acc8c7b0994 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649418272.249736443","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id deployed_model_id = deploy_model.deploy_model( File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 65, in deploy_model response = client.deploy_model( File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ return wrapped_func(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "", line 3, in raise_from google.api_core.exceptions.NotFound: 404 Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.
The quota request is still in progress. I'm going to close this issue. Flaky bot will re-open it if there is still a quota issue.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: 15aaa57c4202a21e2c2fefc00e03bffab7f5ec6c buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649586003.269965779","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model traffic_split={"0": 100}, File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ return wrapped_func(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "", line 3, in raise_from google.api_core.exceptions.NotFound: 404 Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.
Current status on quota request:
We are still working on your request. Rest assured that we are keeping a close eye on this for any changes on your case. Please expect an update from us in the next 5 days or as soon as we receive an update from the engineering team.
Oops! Looks like this issue is still flaky. It failed again. :grimacing:
I reopened the issue, but a human will need to close it again.
commit: 568d19b10e6a2a48892c7a22a8771c43434c40c8 buildURL: Build Status, Sponge status: failed
Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found." debug_error_string = "{"created":"@1649758879.742826862","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.","grpc_status":5}" > The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model traffic_split={"0": 100}, File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 1157, in deploy_model response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ return wrapped_func(*args, **kwargs) File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "", line 3, in raise_from google.api_core.exceptions.NotFound: 404 Model `projects/python-docs-samples-tests/locations/us-central1/models/8785722428534816768` is not found.
Googlers see b/227460894
Note: #6734 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: aa4753979552101612fd6d9def277c9a59d38364 buildURL: Build Status, Sponge status: failed
Test output