abetancurg
commented
2 years ago Login
The first thing I recommend is to check if I have the Cloud SDK already installed. If not, by this url I can download it.
On this shell, is necessarily run the following code to authenticate: gcloud auth login and gcloud auth application-default login
When I am already done with the auth, is quite well check the project out by running the code gcloud config list
Running the URL
What I need to do left is to run the big URL. It's big in this case bcuz the target here is to run it throught Dataflow and not throught DirectRunner.
For what I mentioned above, is too important to keep in mind the tips below:
- If I built 3 custom options it is necessary pass these 3 options by the big URL
- Whether I want to run by Dataflow, it is necessary to pass the following beam options:
-
- --runner DataflowRunner
-
- --project {this project is mentioned when I run gcloud config list}
-
- --region us-central1 -> If I prefer another region I might extract it from here
-
- --temp_location {It is the route in storage where Dataflow paste its data, e.g. gs://ct-pio/tmp/}
-
- --subnetwork {https://www.googleapis.com/compute/v1/projects/{project_id}/regions/{region I chose above}/subnetworks/default} <- I can improve this knowledge by here
- As I will run this pipeline by Dataflow, the files I will use might not be locally, but on the storage. So, it's so usefull to check if actually I can see the routes of the files from the Shell by running the code
gsutil lsand if I want to zoom it, the code isgsutil ls gs://ct-pioe.g. and with these results I can finally fullfill the part of the custom options of the big URL with confidence. - If I want to check more details of the files on storage I can run
gsutil ls -hl gs://ct-pio
One example of the big URL is:
python27 ejercicio2.py --entrada gs://ct-pio/data/el_quijote.txt --salida gs://ct-pio/out/salida.csv --runner DataflowRunner --project contento-bi --region us-central1 --temp_location gs://ct-pio/tmp/ --n-palabras 3 --subnetwork https://www.googleapis.com/compute/v1/projects/contento-bi/regions/us-central1/subnetworks/contento-subnet1Note: After run this URL and the Shell shows a 401 or 403 Error, please read the Login part of this issue.
Pending to write...