Closed snowch closed 2 months ago
Hi, Thank you for sharing this interesting use case. SparkMeasure only captures the metrics provided by the Spark Listener. As you mentioned, it might be worth investigating if this issue can be resolved at the DB plugin level.
Thanks!
I'm hoping to use sparkmeasure to compare performance of reading data from S3 vs reading from an optimised database.
CSV on S3 works as expected:
This reports:
The Database query:
For some reason this is reporting zero bytes read.
Should I be looking into the DB plugin code to debug this? https://github.com/vast-data/vast-db-connectors