Get more workers with Google Cloud's free trial

PolyAI-LDN / conversational-datasets

Large datasets for conversational AI

Apache License 2.0

1.29k stars 167 forks source link

Get more workers with Google Cloud's free trial #78

Open drunkinlove opened 2 years ago

drunkinlove commented 2 years ago

In case this helps anyone: my dataflow job on Google Cloud was restricted to 1 worker, so I added the following argument when launching create_data.py: --num_workers 8 Now the job was restricted to 4 workers instead due to the quota on external IP addresses. So I turned them off with: --no_use_public_ips and enabling Private Google Access for the used region's subnet (us-central1 for me) in the VPC networks section.

Clovin commented 1 year ago

Thanks!

Clovin commented 1 year ago

The max worker num is 20.