search

culturecreates / footlight-aggregator

A tool to inject entities from Artsdata to footlight
0 stars 0 forks source link

create recurring activity to check for errors in the aggregator #132

Closed troughc closed 1 week ago

troughc commented 1 month ago

Please set up a (weekly?) recurring activity to check for errors in all aggregators and diagnose problems as they occur. Also report on status weekly in a daily standup

Also consider adding a notification for users so they know how many events were updated / created and by which aggregator. This aspect could be implemented in https://github.com/culturecreates/footlight-app/issues/827

sahalali commented 3 weeks ago

The aggregator log has been updated. The revised log now appears as follows:Image :

Let's create weekly aggregator reports by downloading filter logs from datadog. This will help us to go through the summary and if needed in detail as well.

Example: https://app.datadoghq.com/logs?query=service%3Afootlight-aggregator%20Import%20statistics&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=desc&viz=stream&from_ts=1730184466062&to_ts=1730789266062&live=true

Created a dashboard for aggregator in data dog and also added a graph to display aggregator summaries. see here

sahalali commented 3 weeks ago

@troughc We can notify users about the status of the aggregator run through the CMS by using the CMS API to send them a message after each run. This will ensure users are informed in CMS. One point that is unclear to me is whether we should notify all admin users in the calendars about the aggregator run summary and the aggregator user. I believe we need to inform all admin users. It should be a summary message like the one in the attached screenshot.

sahalali commented 3 weeks ago

I set up a monitor in Datadog to notify admin@culturecreates.com if any aggregators finish running with at least one error. Image

troughc commented 3 weeks ago

@sahalali comment from @saumier : Please add current practice to wiki and note location here. Thanks

sahalali commented 3 weeks ago

The notes are added to the wiki

troughc commented 2 weeks ago

@sahalali what's next?

sahalali commented 2 weeks ago

I will preset the aggregator-run reports every MONDAY (back day is TUESDAY or the first day of a week) during the standup. I think we can close this ticket as we have a plan.