Design the publishing API

[brayanjuls]

We want to support writing to different sources the generated metrics, the output schema should be ideally standard and the writing mode should be "append only".

[josechudev]

Hi

[brayanjuls]

One Idea to kick off this issue is splitting it on three parts, storage backend, file format and base schemas:

• Storage backend: for this we can consider implementing a local disk backend, object storage(s3) backend and relational db backend.
• File Format: because this library has a focus on analytical workloads we can choose to use table formats like delta for object storage and disk backend.
• Base Schemas: I think initially we can have the following schema as base and in the support defining custom schemas for different metrics. Schema:

  metric_name: String
  event_ts: Timestamp
  system_ts: Timestamp
  value: float. 
  tags: Array<String>

[josechudev]

Hey @brayanjuls, I see you implemented the publish function in metrics.rs, should we move this to core/publishing.rs eventually? I am moving forward with LocalDisk and S3 Backend.