Add functionality & tests to data handling utils - Githubissues

ethereum-optimism / op-analytics

Onchain Data, Utilities, References, and other Analytics on Optimism

104 stars 47 forks source link

Add functionality & tests to data handling utils #931

Open dioptx opened 1 day ago

dioptx commented 1 day ago

Description

Updated write.py in op_coreutils/bigquery with:
- New functions: write_df_to_bq, upsert_partition, refactored overwrite_partitions.
- Enhanced docstrings and exception handling.
- upsert_partition for unique-key-based data upserts.
Added test suite test_write.py covering:
- Initialization, overwrite, upsert, and partition handling.
- Edge cases for empty DataFrames and unique keys in upserts.

Tests

Created tests in test_write.py for BigQuery write operations:
- Valid and invalid date partitioning.
- Upsert with various unique keys.
- Error handling for empty DataFrames and invalid configurations.