search

opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
69 stars 16 forks source link

Hasura => CloudSQL #369

Closed ryscheng closed 7 months ago

ryscheng commented 11 months ago

Hasura can connect directly to BigQuery

https://hasura.io/docs/latest/databases/bigquery/index/

Let's add that connection when it's time.

Need devops solution for configuring Hasura including

davidgasquez commented 11 months ago

If you end up connecting Harusa to BigQuery you might end up with a veeeery large bill (even with caching). :money_mouth_face:

Not entirely sure if this pattern will apply but if;

  1. the dataset changes slowly (once a day, ...)
  2. end users don't require real time data
  3. is less than 1TB

You could export it to files and put something like RoAPI or Dozer on top of it.

Another approach if the data is super small and don't need a GraphQL API would be to generate static JSON files (ala Allo) and make that the API.

ryscheng commented 11 months ago

Yep I agree. I think in the short-term, I was imagining we export to file and import to Postgres https://hasura.io/docs/latest/schema/postgres/postgres-guides/import-data-from-csv/

But we should do whatever is faster/cheaper/easier. Thanks for the recommendations!

davidgasquez commented 8 months ago

Chiming in to share a cool product, Tinybird. It might be a simple way to have both a fast database (Clickhouse) on top of the BigQuery data and an exposed API.

ryscheng commented 7 months ago

Using the Hasura CLI as of https://github.com/opensource-observer/oso/pull/851

which sets this up