observablehq / framework

A static site generator for data apps, dashboards, reports, and more. Observable Framework combines JavaScript on the front-end for interactive graphics with any language on the back-end for data analysis.
https://observablehq.com/framework/
ISC License
2.41k stars 110 forks source link

Importing `@google-cloud/bigquery` fails in JS snippets #1488

Closed benminer closed 3 months ago

benminer commented 3 months ago

@observablehq/framework: 1.9.0 NodeJS: 20.15.0

I got a static data loader working with BigQuery:

// data/bigquery.ts
import { BigQuery } from "@google-cloud/bigquery";

const bigquery = new BigQuery();

export const runQuery = async (query) => {
  const [rows] = await bigquery.query(query);
  return rows;
};

This works fine loading static data, like countries, from the database:

// data/countries.json.ts
import { runQuery } from "../bigquery";

process.stdout.write(
  JSON.stringify(
    await runQuery(`
SELECT country FROM $my-bg-table
`))
);

However, in the page I am trying to create, it needs to allow a viewer to pick a country from a dropdown. The dropdown works just fine when running, but I can't figure out how to do a dynamic query using my new bigquery.ts file. When I import in a "``js" tag, I getTypeError: Failed to resolve module specifier 'data/bigquery'. Running this with atstag simply displays the code, but doesn't run anything. Installing@google-cloud/bigqueryvianpm ialso throws:TypeError: Failed to resolve module specifier @google-cloud/bigquery - which doesn't make sense, since it's installed. Using the npm: prefix throws CJS bundle errors from esbuild.

// my dashboard.md
```js
const countries = await FileAttachment("data/countries.json").json();
\```

```js
const countrySelect = Inputs.select(
  countries.map((obj) => obj.country),
  {
    label: "Select Country",
    value: "ES",
  }
);
display(countrySelect); // this works!
\```

```js
import { runQuery } from "data/bigquery"; // this throws
\```

I've tried importing the BigQuery library directly in a js block, but that leads to bundle errors from Rollup: Failed to bundle using Rollup v2.79.1: the file imports a not supported node.js built-in module "fs".

```js
import { BigQuery } from "npm:@google-cloud/bigquery"; // throws
import { BigQuery } from "@google-cloud/bigquery"; // throws
const { BigQuery } = await import('@google-cloud/bigquery') // throws
const bigquery = new BigQuery();
const data = await bigquery.query(`
SELECT country, ymd, hour
FROM \`my-bg-table\`
WHERE ymd >= "2024-06-20"
and country= ${countrySelect}
and hour is not null
  and region is null
order by ymd, hour
`);
display(data);
\```

I understand the bundle errors here, but there has got to be a way to run a dynamic BigQuery query in a snippet, right? I am new to Observable overall, so there may be something obvious here. Is there anyway to use the built in sql tag with connected data sources in my account, so I could just do a query as I would in a Notebook?

Any help is appreciated, thanks.