search

tusharchou / local-data-platform

python library for iceberg lake house on your local
MIT License
8 stars 5 forks source link

0.1.1 Create a Google BigQuery Client
 #19

Open tusharchou opened 1 month ago

tusharchou commented 1 month ago

Use Python's BigQuery API to pull the data from the Near Protocol dataset. For example, this code queries the transactions table and retrieves the data for processing:

Create a BigQuery client

from google.cloud import bigquery client = bigquery.Client()

Define your query to pull data from Near Protocol dataset

query = """ SELECT * FROM bigquery-public-data.crypto_near.transactions LIMIT 1000; """

Run the query and convert the result to a Pandas DataFrame

query_job = client.query(query) transactions_df = query_job.to_dataframe()

This retrieves the transaction data, which you can load into Apache Iceberg for storage and further processing.

tusharchou commented 1 month ago

@mrutunjay-kinagi picking this up!

tusharchou commented 1 month ago

Reviewing now

mrutunjay-kinagi commented 4 weeks ago

Reopening this issue to align with new approach as mentioned in #51

mrutunjay-kinagi commented 4 weeks ago

I'll be working on this. @tusharchou @redpheonixx

tusharchou commented 3 weeks ago

Hi @mrutunjay-kinagi Thanks you for picking this up.

We need a feature to login our credentials as a JSON template for GCP.

So that I can demo to the community

tusharchou commented 3 weeks ago

@mrutunjay-kinagi please create a pr to demo this capability

mrutunjay-kinagi commented 3 weeks ago

Pushed changes in PR #56

tusharchou commented 3 weeks ago

Thank you @mrutunjay-kinagi for working on this, and closing it

redpheonixx commented 2 weeks ago

image

mrutunjay-kinagi commented 1 week ago

This issue is resolved in PR #86 @tusharchou @redpheonixx