GoogleCloudPlatform / bigquery-utils

Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.
https://cloud.google.com/bigquery/
Apache License 2.0
1.13k stars 283 forks source link

Documentation for new users is difficult to follow #280

Open kerryleith opened 2 years ago

kerryleith commented 2 years ago

I've never used SQL, but I have a need to here, as I want to implement ts_tumble in BigQuery to reduce my data for graphing.

I tried following the user guide in Cloudshell Terminal, but I got stuck at:

user_name@cloudshell:~/cloud-console-sample-build (project_name)$ gcloud builds submit . --config=deploy.yaml --substitutions _BQ_LOCATION=US ERROR: (gcloud.builds.submit) Unable to read file [deploy.yaml]: [Errno 2] No such file or directory: 'deploy.yaml'

I suspected it was because I hadn't cloned git to my project, so I ran: git clone https://github.com/GoogleCloudPlatform/bigquery-utils.git

But got the same error for the deploy command, and still haven't figured out how to run the UDF in the SQL workspace.

Apologies for the broad issue, but perhaps more users could benefit from a documentation upgrade.

Thanks!

danieldeleo commented 2 years ago

Thanks for the feedback, @kerryleith. I'll improve the UDF docs.

danieldeleo commented 2 years ago

In the meantime, try running the same command you tried but within the udfs/ directory.

kerryleith commented 2 years ago

Thanks Daniel,

I know writing docs isn't the most exciting aspect of developing a project, but it always helps :)

I'm preparing a step-by-step beginners guide to describe the process of collecting data from an Arduino through to displaying it in DataStudio. So far I've avoided installing any software on a local machine, since each CLI or software tool has its own set of commands, and a challenging learning curve that can be an intimidating barrier to progression. To that end, I don't even know how to change to the directory.. I know, my bad, but I don't know if you mean on the server, or on my machine.

I'd like to maintain the workflow, and present the console option in the guide.

What would be fantastic, would be a worked example, or screenshot to expand on the instruction below: "BigQuery Console: Just paste the SQL UDF body in the console and execute."

Thanks so much

kerryleith commented 2 years ago

Hi Daniel,

I worked on this for a while, and I think the issue was that my data is not stored in on the Default US server. I think the Git UDF's only operate on data on the default server. I managed to solve the problem by manually copying the UDF and specifying a location to create the function e.g.

CREATE OR REPLACE FUNCTION 'my-project-id.my_data_set'.tumble_interval(input_ts TIMESTAMP, tumble_seconds INT64) RETURNS TIMESTAMP AS ( IF ( tumble_seconds > 0, TIMESTAMP_SECONDS(DIV(UNIX_SECONDS(input_ts), tumble_seconds) * tumble_seconds), NULL ) );