Create metadata reporting capability

teragrep / pth_10

Data Processing Language (DPL) translator for Apache Spark

GNU Affero General Public License v3.0

0 stars 2 forks source link

Create metadata reporting capability #243

Closed kortemik closed 4 months ago

kortemik commented 5 months ago

Description Metadata reporting offers ability to report how much data (compressed, uncompressed, rows) is stored per index/sourcetype/app

Use case or motivation behind the feature request Customer request for internal reporting of data volumes

Related issues Exporting via #150

Additional context

eemhu commented 4 months ago

@kortemik : Should this be under the teragrep command, for example | teragrep exec create report [by=field] [compressed=bool] [uncompressed=bool] [rows=bool] where by= works like groupBy (index, sourcetype, app), and the remaining params allow choosing if certain metrics are included or not?

eemhu commented 4 months ago

Design meeting notes:

Add to pth-06 datasource as the tables contain a lot of data
Add additional metadata to pre-existing metadata object
Use a parameter / flag to indicate "metadata fetching mode" in datasource (Create separate InputPartitionReader for metadata)
Use logicalStatement with special flag to indicate metadata fetching mode, see search command
"| teragrep get archive summary"

eemhu commented 4 months ago

submitted pth-06 and pth-10 PRs