aptos-labs / aptos-indexer-processors

Set of core processors that index data on the Aptos blockchain
https://aptos.dev/indexer/indexer-landing
60 stars 72 forks source link

[SDK-parquet] add parquet version tracker #609

Closed yuunlimm closed 1 week ago

yuunlimm commented 2 weeks ago

Description

1. Added Parquet Version Tracker Functionality

2. Schema and Table Handling Updates

3. Tests updated

4. General Code Improvements

Enhanced comments for better readability and maintainability.

Test Plan Screenshot 2024-11-15 at 12 35 53 PM

Screenshot 2024-11-15 at 11.38.50 AM.png

backfill DQ check on number of rows for Move_resources Screenshot 2024-11-18 at 11 12 27 AM Screenshot 2024-11-18 at 11 12 24 AM

yuunlimm commented 2 weeks ago

This stack of pull requests is managed by Graphite. Learn more about stacking.

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 34.46712% with 289 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@5160738). Learn more about missing BASE report.

Files with missing lines Patch % Lines
rust/sdk-processor/src/parquet_processors/mod.rs 0.0% 69 Missing :warning:
...r/src/steps/common/parquet_version_tracker_step.rs 0.0% 65 Missing :warning:
...rc/parquet_processors/parquet_default_processor.rs 0.0% 44 Missing :warning:
...ust/sdk-processor/src/steps/common/gcs_uploader.rs 0.0% 36 Missing :warning:
...ocessor/src/steps/common/processor_status_saver.rs 57.5% 14 Missing :warning:
...k-processor/src/processors/monitoring_processor.rs 0.0% 10 Missing :warning:
...r/src/processors/account_transactions_processor.rs 50.0% 5 Missing :warning:
rust/sdk-processor/src/processors/ans_processor.rs 50.0% 5 Missing :warning:
.../sdk-processor/src/processors/default_processor.rs 50.0% 5 Missing :warning:
...t/sdk-processor/src/processors/events_processor.rs 50.0% 5 Missing :warning:
... and 8 more
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #609 +/- ## ====================================== Coverage ? 48.6% ====================================== Files ? 186 Lines ? 24911 Branches ? 0 ====================================== Hits ? 12129 Misses ? 12782 Partials ? 0 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


🚨 Try these New Features: