search

GoogleCloudPlatform / professional-services

Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.
Apache License 2.0
2.83k stars 1.33k forks source link

BQ dataset used in dataflow example doesn't exist #527

Open dylanmpeck opened 4 years ago

dylanmpeck commented 4 years ago

The python-dataflow-example.example_data BigQuery dataset used in the cookbook batch-examples of dataflow-examples code is returning a 404 and all operations with code that references it are failing.

Did this dataset change in some way recently?

Link to two files where I know dataset is referenced: https://github.com/GoogleCloudPlatform/professional-services/blob/master/examples/dataflow-python-examples/batch-examples/cookbook-examples/pipelines/data_enrichment.py https://github.com/GoogleCloudPlatform/professional-services/blob/master/examples/dataflow-python-examples/batch-examples/cookbook-examples/pipelines/data_lake_to_mart.py

prathapreddy123 commented 4 years ago

We recently moved the code around (without any logic changes) to different directory as part of restructuring. Can you verify if call is happening to right path.

dylanmpeck commented 4 years ago

We are using a forked version of repo that is still in the state of before the directories were restructured. This is for a Qwiklabs lab that uses this code.

I believe the path is right, but here's a picture with error message: Job failed

To add more info, we only started getting reports of this error since yesterday.

prathapreddy123 commented 4 years ago

Based on error message project python-dataflow-example seems to be missing. Can you cross check if project and corresponding BQ table exists

dylanmpeck commented 4 years ago

I don't think so. In BigQuery within my "qwiklabs..." project, I'm unable to locate the table and I'm unable to do simple queries on the python-dataflow-example project + table. Always getting a "doesn't exist" related error.

Is the table publicly hosted somewhere? Or, is the table supposed to be generated by the example code?

I had thought both examples were using a public table as a 2nd source based on how the table is referenced.

empz commented 4 years ago

The data doesn't not exist or is not set as public.

gsutil ls -r gs://python-dataflow-example/data_filesreturns BucketNotFoundException: 404

tommo245 commented 3 years ago

Hi - Also not finding the necessary template, getting this error.

File "/path/to/lib/python3.8/site-packages/google/cloud/_http.py", line 438, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/storage/v1/b/python-dataflow-example?projection=noAcl&prettyPrint=false: Not Found
agold-rh commented 1 year ago

@joshmcginley @lcaggio Do you have any input here, or should this be closed as stale?