Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.62k stars 586 forks source link

Failure to run mrjob on dataproc #2216

Open BradHolmes opened 2 years ago

BradHolmes commented 2 years ago

I set GOOGLE_APPLICATION_CREDENTIALS env variable properly, and am running a simple mrjob with -r dataproc option. However, it says

google.api_core.exceptions.Unknown: None Stream removed

While calling

self.cluster_client.get_cluster()

I see that the v1beta2 API has been deprecated in favor of v1. Does the dataproc plugin need a re-write to adopt it? Anything I can do to help?