search

GoogleCloudPlatform / python-docs-samples

Code samples used on cloud.google.com
Apache License 2.0
7.34k stars 6.39k forks source link

dataproc.snippets.submit_job_test: test_submit_job failed #9513

Closed flaky-bot[bot] closed 11 months ago

flaky-bot[bot] commented 1 year ago

Note: #8893 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: 161f9cb36ecded78327abeabb81f2ee14254a862 buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 52, in setup_teardown
    operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
flaky-bot[bot] commented 1 year ago

Looks like this issue is flaky. :worried:

I'm going to leave this open and stop commenting.

A human should fix and close this.

When run at the same commit (161f9cb36ecded78327abeabb81f2ee14254a862), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

rsamborski commented 1 year ago

Looks similar to #9510

leahecole commented 1 year ago

I just added a retry for data science onramp in #9496 - we can likely use the same approach here - same problem, same product

flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: f735455f1faaf7a149c7d16f300bd02cace40302 buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.ALREADY_EXISTS
    details = "Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-6113358d-9ce4-4490-a198-ba4c01377645"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.197.95:443 {grpc_message:"Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-6113358d-9ce4-4490-a198-ba4c01377645", grpc_status:6, created_time:"2023-04-15T00:13:23.320016807+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 70, in setup_teardown
    setup()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/workspace/dataproc/snippets/submit_job_test.py", line 52, in setup
    request={"project_id": PROJECT_ID, "region": REGION, "cluster": CLUSTER}
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 635, in create_cluster
    metadata=metadata,
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 354, in retry_wrapped_func
    on_error=on_error,
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.AlreadyExists: 409 Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-6113358d-9ce4-4490-a198-ba4c01377645
leahecole commented 1 year ago

Closing this to see if #9666 helped

flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: e0cb46783f904f7490b5cf4c7a3b8db9aea536eb buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.ALREADY_EXISTS
    details = "Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-3c1ccf30-c4b0-43c9-92d5-e46d52e46339"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.112.95:443 {created_time:"2023-04-20T13:04:59.913402956+00:00", grpc_status:6, grpc_message:"Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-3c1ccf30-c4b0-43c9-92d5-e46d52e46339"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 70, in setup_teardown
    setup()
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/workspace/dataproc/snippets/submit_job_test.py", line 51, in setup
    operation = cluster_client.create_cluster(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 631, in create_cluster
    response = rpc(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.AlreadyExists: 409 Already exists: Failed to create cluster: Cluster projects/python-docs-samples-tests/regions/us-central1/clusters/py-sj-test-3c1ccf30-c4b0-43c9-92d5-e46d52e46339
leahecole commented 1 year ago

Given that we added retries for certain transient service errors, we need to check for an "already exists" and ignore on it, making sure we always teardown

flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: 6d04d56c1bd822ab62d8ab65a3b00605b8325c44 buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 79, in cluster_name
    setup_cluster(cluster_client, curr_cluster_name)
  File "/workspace/dataproc/snippets/.nox/py-3-9/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/workspace/dataproc/snippets/submit_job_test.py", line 55, in setup_cluster
    operation.result()
  File "/workspace/dataproc/snippets/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 The resource 'projects/python-docs-samples-tests/regions/us-central1/subnetworks/default' is not ready
flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: 62865642041ade48650ae0debcc3a80898c86def buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.FAILED_PRECONDITION
    details = "Multiple validation errors:
 - No agent on master node(s) reported to be be active
 - Unable to submit job, cluster 'py-sj-test-74559d8c-3b83-42a7-a70b-218ce75a93d1' is in state ERROR and cannot accept jobs."
    debug_error_string = "UNKNOWN:Error received from peer ipv4:173.194.196.95:443 {created_time:"2023-05-02T13:03:19.051827077+00:00", grpc_status:9, grpc_message:"Multiple validation errors:\n - No agent on master node(s) reported to be be active\n - Unable to submit job, cluster \'py-sj-test-74559d8c-3b83-42a7-a70b-218ce75a93d1\' is in state ERROR and cannot accept jobs."}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 91, in test_submit_job
    submit_job.submit_job(PROJECT_ID, REGION, cluster_name)
  File "/workspace/dataproc/snippets/submit_job.py", line 53, in submit_job
    operation = job_client.submit_job_as_operation(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/cloud/dataproc_v1/services/job_controller/client.py", line 683, in submit_job_as_operation
    response = rpc(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/workspace/dataproc/snippets/.nox/py-3-10/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.FailedPrecondition: 400 Multiple validation errors:
 - No agent on master node(s) reported to be be active
 - Unable to submit job, cluster 'py-sj-test-74559d8c-3b83-42a7-a70b-218ce75a93d1' is in state ERROR and cannot accept jobs.
flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: 43304bc95e0b2f0ac73fb061dff13d3d788a0b48 buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 95, in test_submit_job
    assert response.status.state == dataproc.ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1683576209\n  nanos: 594460000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1683576209\n  nanos: 594460000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-sj-test-da8f9bcc-a8fb-4103-8bc7-bcd19e11ac8e"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-sj-test-da8f9bcc-a8fb-4103-8bc7-bcd19e11ac8e-m"\n    image_uri...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "291c1410-be24-42a4-863f-714e8ed6fe02"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-sj-test-da8f9bcc-a8fb-4103-8bc7-bcd19e11ac8e"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1683576209\n    nanos: 594460000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1683576207\n    nanos: 111009000\n  }\n}\ncluster_uuid: "291c1410-be24-42a4-863f-714e8ed6fe02"\n.status
 +  and    = .RUNNING
 +    where  = .State
 +      where  = dataproc.ClusterStatus
leahecole commented 1 year ago

I actually think this is WAI for now - it failed when the cluster was in an ERROR state. I'm goign to close to see how long it takes to reopen.

flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: 0b7616fd3a3ecd906e48a6932eb3ce73007b738d buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 95, in test_submit_job
    assert response.status.state == dataproc.ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1683801581\n  nanos: 390865000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1683801581\n  nanos: 390865000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-sj-test-6060027e-8363-4d5c-a8cd-cbcb75436a25"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-sj-test-6060027e-8363-4d5c-a8cd-cbcb75436a25-m"\n    image_uri...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "ae5ecaf1-d078-473b-a2e5-0191744c50d3"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-sj-test-6060027e-8363-4d5c-a8cd-cbcb75436a25"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1683801581\n    nanos: 390865000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1683801578\n    nanos: 885615000\n  }\n}\ncluster_uuid: "ae5ecaf1-d078-473b-a2e5-0191744c50d3"\n.status
 +  and    = .RUNNING
 +    where  = .State
 +      where  = dataproc.ClusterStatus
flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: 27b904b0a411023221b8ba3173b45f351d52db7c buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 95, in test_submit_job
    assert response.status.state == dataproc.ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1685447246\n  nanos: 956433000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1685447246\n  nanos: 956433000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-sj-test-fd92d5f6-5d10-4f31-b8f5-9922737f68b1"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n  }\n  master_config {\n    num_instances: 1\n    instance_names: "py-sj-test-fd92d5f6-5d10-4f31-b8f5-9922737f68b1-m"\n    image_uri...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "ddbaaa16-6643-457f-8e2c-96c8b7078e15"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-sj-test-fd92d5f6-5d10-4f31-b8f5-9922737f68b1"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1685447246\n    nanos: 956433000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1685447239\n    nanos: 874557000\n  }\n}\ncluster_uuid: "ddbaaa16-6643-457f-8e2c-96c8b7078e15"\n.status
 +  and    = .RUNNING
 +    where  = .State
 +      where  = dataproc.ClusterStatus
flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: e02c4b623b03554df22e1b2c65a77a76501963d0 buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 113, in test_submit_job
    assert response.status.state == dataproc.ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1690290333\n  nanos: 954908000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1690290333\n  nanos: 954908000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-sj-test-c0139978-9bbc-4793-af76-e93edca5aab3"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n    service_account_scopes: "https://www.googleapis.com/auth/monitoring.write"\n  }\n  master_config {\n    num_instances: 1\n    in...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "3f73f9f5-71b1-487b-93f3-a20cc3df99c5"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-sj-test-c0139978-9bbc-4793-af76-e93edca5aab3"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1690290333\n    nanos: 954908000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1690290331\n    nanos: 262217000\n  }\n}\ncluster_uuid: "3f73f9f5-71b1-487b-93f3-a20cc3df99c5"\n.status
 +  and    = .RUNNING
 +    where  = .State
 +      where  = dataproc.ClusterStatus
leahecole commented 1 year ago

Closing to see if #10527 helped

flaky-bot[bot] commented 1 year ago

Oops! Looks like this issue is still flaky. It failed again. :grimacing:

I reopened the issue, but a human will need to close it again.

commit: e440097f58ea5d5ddb883b7b6099c7bec0ee3c7a buildURL: Build Status, Sponge status: failed

Test output
Traceback (most recent call last):
  File "/workspace/dataproc/snippets/submit_job_test.py", line 113, in test_submit_job
    assert response.status.state == dataproc.ClusterStatus.State.RUNNING
AssertionError: assert  == 
 +  where  = state: ERROR\nstate_start_time {\n  seconds: 1692968912\n  nanos: 995776000\n}\n.state
 +    where state: ERROR\nstate_start_time {\n  seconds: 1692968912\n  nanos: 995776000\n}\n = project_id: "python-docs-samples-tests"\ncluster_name: "py-sj-test-b9c0c1d7-5dcd-4612-8ea1-37082256fa18"\nconfig {\n  config_bucket: "dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1"\n  temp_bucket: "dataproc-temp-us-central1-1012616486416-3kbzlm1c"\n  gce_cluster_config {\n    zone_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/zones/us-central1-f"\n    network_uri: "https://www.googleapis.com/compute/v1/projects/python-docs-samples-tests/global/networks/default"\n    service_account_scopes: "https://www.googleapis.com/auth/bigquery"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.admin.table"\n    service_account_scopes: "https://www.googleapis.com/auth/bigtable.data"\n    service_account_scopes: "https://www.googleapis.com/auth/cloud.useraccounts.readonly"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.full_control"\n    service_account_scopes: "https://www.googleapis.com/auth/devstorage.read_write"\n    service_account_scopes: "https://www.googleapis.com/auth/logging.write"\n    service_account_scopes: "https://www.googleapis.com/auth/monitoring.write"\n  }\n  master_config {\n    num_instances: 1\n    in...   properties {\n      key: "distcp:mapreduce.map.memory.mb"\n      value: "768"\n    }\n    properties {\n      key: "distcp:mapreduce.map.java.opts"\n      value: "-Xmx576m"\n    }\n    properties {\n      key: "core:hadoop.ssl.enabled.protocols"\n      value: "TLSv1,TLSv1.1,TLSv1.2"\n    }\n    properties {\n      key: "core:fs.gs.metadata.cache.enable"\n      value: "false"\n    }\n    properties {\n      key: "core:fs.gs.block.size"\n      value: "134217728"\n    }\n    properties {\n      key: "capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy"\n      value: "fair"\n    }\n  }\n  endpoint_config {\n  }\n}\nlabels {\n  key: "goog-dataproc-location"\n  value: "us-central1"\n}\nlabels {\n  key: "goog-dataproc-cluster-uuid"\n  value: "4d871990-3a90-4568-964f-17a9f8a4deca"\n}\nlabels {\n  key: "goog-dataproc-cluster-name"\n  value: "py-sj-test-b9c0c1d7-5dcd-4612-8ea1-37082256fa18"\n}\nlabels {\n  key: "goog-dataproc-autozone"\n  value: "enabled"\n}\nstatus {\n  state: ERROR\n  state_start_time {\n    seconds: 1692968912\n    nanos: 995776000\n  }\n}\nstatus_history {\n  state: CREATING\n  state_start_time {\n    seconds: 1692968910\n    nanos: 362880000\n  }\n}\ncluster_uuid: "4d871990-3a90-4568-964f-17a9f8a4deca"\n.status
 +  and    = .RUNNING
 +    where  = .State
 +      where  = dataproc.ClusterStatus