Data import status overview

[tovari]

There are a lot of new external data sources that GO uses to fetch data and present on Country pages, Risk, Surge, Appeals... We need to be able to have an overview about all the availability of all these sources and a log of data intake attempts and results. First we could have an overview table and log table(s). The overview would need an api access as well.

We need to know:

• Data source name
• Latest import time
• Latest import status
• Latest import error code and message
• Latest successful import time

Logs needed about import attempt times, and results for the last 60 days.

[samshara]

Proposed Workflow

1. Documentation of scheduled tasks

   First, we will prepare a document to track all scheduled tasks. This process will be separated into two separate documents.

   • An Excel document that will list all the scheduled tasks with necessary metadata information. The Excel file will be formatted as follows

   Task Name	Description	Frequency	Command	Working Directory	Enabled	Source	Validated
   Task 1	A simple task	Daily	python script.py	app	True	https://data.ifrc.org/FDRS/	False

   • A markdown file with additional details. This could include
     • Impacted entities ( Reflection, Tables, Triggers)
     • Complexity of the script?
     • Does it need refactoring in the future?
     • What is the source data type?
     • Does it require credentials?

2. Documentation of unscheduled tasks

   We will do the same for scripts that run through management commands, translations, and exports

3. Monitor tasks

   We can monitor the scheduled task using external monitoring applications. We must discuss the potential tools that we could use.

   cc: @tovari @thenav56 @k9845

[nnjemie]

Documentation of scheduled tasks: https://docs.google.com/document/d/1aXd6JfhLa2io0ndkoxdCA1ytXTV3ReQfMj7xtl6jbgg/edit#heading=h.re32euektpkx

Scheduled tasks that will go into api: https://docs.google.com/spreadsheets/d/1FRZZPhx1tRj-FMRVAAtaaUZ687E-SWR_BMtPQ2Zagtg/edit#gid=0

cc: @tovari @udaynwa

[samshara]

@tovari , Here are the detailed reports on scheduled and unscheduled tasks Unscheduled Tasks Report Scheduled Tasks Report

cc: @nnjemie

[udaynwa]

Documentation of Risk Watch's tasks: https://docs.google.com/document/d/1_lUJuMsy5sBZ8GGm7oEvU1vNMp0aI8A8hTCCtuIcmaQ/edit?usp=sharing

Report of Risk Watch's tasks: https://docs.google.com/spreadsheets/d/1idisC_H7JirraT8tRQbVlfUazKoGFU63/edit?usp=sharing&ouid=113765549790989507676&rtpof=true&sd=true

cc @tovari @nnjemie @samshara @frozenhelium