search

IATI / refresher

A Python application which has the responsibility of tracking IATI data from around the Web and refreshing the core IATI software's data stores
GNU Affero General Public License v3.0
2 stars 0 forks source link

Can we have more regularly updated Prom metrics? #355

Open odscjames opened 1 month ago

odscjames commented 1 month ago

(This issues looks in detail at one stage only, but I think it applies to all stages).

As a sys admin doing operations I want to see more regularly updated metrics so that I can monitor the system with accurancy.

I'm currently running an operation to resolerize all documents so ~9000. This operation takes hours. I want to see detailed metrics of how far through the operation is so I can monitor performance, and see the effects of any changes I make (such as giving the system more resources half way through).

However, the current stats this software gives to Solr are only updated at the start of the main loop for solrize, not during. https://github.com/IATI/refresher/blob/develop/src/library/solrize.py#L432

This means that what Promethous sees is:

PROM-GRAPH

I'd like the stats to be updated during the loop, so I see the graph running down over time.

(Ideally I'd like the stats to be updated totally independently of the loop, so that if there is a problem with the main loop the stats still continue to be updated - tho in this case that doesn't matter, as if the loops break "documents to process" won't go down! But maybe it would in the case of other stats)

Acceptance criteria

  1. Functional Criteria (Product features or functions that developers must implement to enable users to accomplish their tasks) 1.1. Stats should update during operations

  2. Non-functional Criteria (Include availability, maintainability, performance, reliability, scalability, security, and usability criteria) 2.1. The updating of stats shouldn't break normal work loop 2.2. Ideally a broken work loop wouldn't stop stats updating (tho ideally a broken work loop would raise an alarm somehow somewhere too!)

odscjames commented 1 month ago

Thought later on: It would be good to get the metrics from the database directly, as then we get a record as new tasks come in as well as tasks are processed.