search

qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

Fix and expose direct data read metric #409

Closed sopel39 closed 4 years ago

shubhamtagra commented 4 years ago

Its needs a overhaul of the metrics system. Something of this sorts need to be done to cleanup current code:

RemoteRRC
    Metric updated: read-from-source-and-cached
        extra read from source due to block alignment
        total read from source

CachedRRC
    Metric updated: read-from-local-node-cache

DirectRRC
    Metric updated: read-from-source-but-not-cached

NonLocalRRC
    Metric updated: read-from-non-local-node
        For NonLocalRRC, there can be a read from source at BookKeeper

NonLocalRC
    Direct + NonLocal

FileDownloadRC
    Do not update metrics. Relevant metrics are updated in BookKeeper as mentioned next

BookKeeper
    Data read from Source during read-through mode: READ_THROUGH_DOWNLOADED_MB_COUNT
    Data read from Source during async mode: ASYNC_DOWNLOADED_MB_COUNT

For the cluster:
(1) Total data read from Source = (read-from-source-and-cached) + (read-from-source-but-not-cached) + READ_THROUGH_DOWNLOADED_MB_COUNT + ASYNC_DOWNLOADED_MB_COUNT
(2) Total data read from local caches = (read-from-local-node-cache)
(3) Total data read from non-local caches = (read-from-non-local-node) - READ_THROUGH_DOWNLOADED_MB_COUNT

Hit rate = ((2) + (3)) / ((1) + (2) + (3))

Additional metrics that can be reported:
    - Extra data read due to block alignment
    - Per RRC metrics