tusharchou / local-data-platform

python library for iceberg lake house on your local
3 stars 1 forks source link

0.1.1 Create a Google BigQuery Client
 #19

Open tusharchou opened 11 hours ago

tusharchou commented 11 hours ago

Use Python's BigQuery API to pull the data from the Near Protocol dataset. For example, this code queries the transactions table and retrieves the data for processing: python

from google.cloud import bigquery

Create a BigQuery client

client = bigquery.Client()

Define your query to pull data from Near Protocol dataset

query = """ SELECT * FROM bigquery-public-data.crypto_near.transactions LIMIT 1000; """

Run the query and convert the result to a Pandas DataFrame

query_job = client.query(query) transactions_df = query_job.to_dataframe()

This retrieves the transaction data, which you can load into Apache Iceberg for storage and further processing.

tusharchou commented 11 hours ago

@mrutunjay-kinagi picking this up!

tusharchou commented 6 hours ago

Reviewing now