search

jrossthomson / bigquery-utils

Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.
https://cloud.google.com/bigquery/
Apache License 2.0
0 stars 0 forks source link

Fix local testing for jslibs, enable local jslibs bundling #5

Closed imathews closed 3 years ago

imathews commented 3 years ago

This PR solves two issues:

  1. The {JS_BUCKET} placeholder wasn't being replaced during local testing (only during deployment and CI builds). This updates test_create_udfs.py to replace all references with either the _JS_BUCKET environment variable, or the production bqutils GCS path (gs://bqutil-lib/bq_js_libs)
  2. There was previously no way to build js_libs locally, which prevents local testing if we add a new library or update an existing library's version. The piping was largely there in existing cloud build scripts; I pulled out only the steps necessary to bundle the js_libs files as new script.

The documentation in udfs/js_libs/README.md has been updated accordingly

imathews commented 3 years ago

@jrossthomson this somehow got bundled with a few commits from the upstream repo — I assume it’s fine to bring these in so that our fork is up-to-date, but let me know if you’d like me to put together a cleaner PR with just my changes.

jrossthomson commented 3 years ago

Merged.

jrossthomson commented 3 years ago

I think the new doc info you have added needs to be added to the contributing section also. Any UDF that uses js_lib will fail if the cloud build is not run, probably confusing people. I'll add that to the contributing.md.

imathews commented 3 years ago

I think the new doc info you have added needs to be added to the contributing section also. Any UDF that uses js_lib will fail if the cloud build is not run, probably confusing people. I'll add that to the contributing.md.

Sounds good. FYI, it'll only fail if they've also added a new js_lib. Otherwise, their UDF will just reference the deployed bucket, which also works fine