Dataflow job is failing

[hrlimaye]

error:

1. Startup of the worker pool in zone us-central1-a failed to bring up any of the desired 1 workers. The project quota may have been exceeded or access control policies may be preventing the operation; review the Stackdriver Logging "GCE VM Instance" log for diagnostics

2. Workflow failed. Causes: Internal Issue (ff80c197c8903cfc): 82159483:17

Any help is appreciated.

[dshetty312]

I am facing the same issue. Any luck?

Regards, DS

[EricBeach]

Can you please clarify the exact steps you're taking to reproduce this, including which the exact commands you entered (full CLI, including flags), the exact version of the code you're running, the environments you're running this in, etc.?

[dshetty312]

I am running the dataflow sample program in GCP.

Command:

mvn -Pdataflow-runner compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--project=<> --stagingLocation= --output=

--runner=DataflowRunner --serviceAccount= --subnetwork= --region=us-east1"

Dataflow SDK version: 2.16.0

Error seen:

SEVERE: 2019-12-10T04:07:24.803Z: Startup of the worker pool in zone us-east1-b failed to bring up any of the desired 1 workers. The project quota may have been exceeded or access control policies may be preventing the operation; review the Stackdriver Logging "GCE VM Instance" log for diagnostics. Dec 10, 2019 4:07:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2019-12-10T04:07:24.827Z: Workflow failed. Causes: Internal Issue (288942aeaaa59265): 82159483:17 Dec 10, 2019 4:07:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process

I dont see any quota problem with the subnet used.

[EricBeach]

Looking at your command, you are not executing Pontem code but rather the standard Apache Beam demo code.

Based upon "the project quota may have been exceeded or access control policies may be preventing the operation", I suspect you have two problems: (1) lack of project quota (2) you have not granted the appropriate permissions (i.e., if you're using a service account, you've not granted the appropriate permissions to that account).

I would suggest engaging the Apache Beam help community.

[dshetty312]

Thanks I got the problem solved.

[jingc-ac]

@dshetty312 I have the same problem, how did you solve it? thx

[dshetty312]

The service account used did not had right permissions

[jrichard8]

Hello, what was the necessary permissions ?

[dshetty312]

@jrichard8 - The service account was missing the ComputeViewer role.

[rainergo]

I had the same problem (zone: europe-west3) and solved it by: 1) Adjusting "Worker IP Address Configuration" to private (as my network is private) and setting the "Network" to my network name 2) Creating a folder "tmp" in my gs bucket (Cloud Storage) as this is referenced in "Temporary location *"

[RoopeshNagiri]

@rainergo can you please let me know how to setup Adjusting "Worker IP Address Configuration" to private (as my network is private) and setting the "Network" to my network name this