Create metrics for displaying the number of document yet-to-be-synced in days terms

[harshad16]

Problem statement

As the document-sync-job, sync documented results to the storage, the idea is to calculate the yet-to-be-synced documents. so we can monitor the results and adjust the scheduled time of the document-sync-job accordingly.

High-level Goals

• Gather the metrics in regard to yet-to-be-synced documented results.

Proposal description

• Calculate the pending documents to be synced by job and update metrics with that number.

[VannTen]

I don't exactly see how we could have "yet-to-be-synced" documents in fact.

Do you mean during the run on the job, or between runs ?

[VannTen]

So resuming the sig discussion from yesterday (@harshad16 whenever you have time could you update the issue description ?) :

• the yet-to-be synced metric would be in a different process (metrics exporter)
• content = basically set difference (source - dest)

Also, I'd like to propose some alternatives:

• I'm not sure, but does ceph expose metrics on the number of documents stocked ? we could use that metric (for the stage ceph and the prod ceph) directly if that's the case and it's rich enough
• Regarding the the speed of our document-sync, I think we can already compute them from the existing metrics by using prometheus aggregation functions (rate or something)

[VannTen]

/triage needs-information

[codificat]

/sig observability

[VannTen]

I think this should be blocked until we finish #37 After that I think we'll have a clearer picture.

[codificat]

/assign @harshad16

[sesheta]

Issues needing reporter input close after 60d,

If there is new input, reopen with /reopen.