Resource Health BB requirements clarification

[dovydas-an]

Perform requirements clarification with other BB developing teams and stakeholders (end users) to clarify checking mechanisms/types and plans for their implementation. In addition, keep track of further development of other BBs to perform iterative requirements refinement of Resource Health BB.

[dovydas-an]

Further requirements clarifications have been performed with APEx and EarthCODE stakeholders (one virtual discussion and additional communication by email).

In terms of checking mechanisms that would be needed, expectations for the following types of functionality have been identified so far (the following list also includes types of functionality that has already been identified as needed in the SoW):

• Ability to execute checks for the following types of resources:
  • Hosted algorithms
  • Hosting environments
  • Third party backends
  • Published datasets
  • ML models
  • Dashboards
• Types of checks:
  • Ability to verify if a resource is visible/callable.
  • Ability to verify if a resource satisfies certain conventions (e.g. publishes a specification that satisfies a set of linting rules).
  • Ability to verify if a resource (specifically Hosting environment) is callable for random spatial/temporal locations (ability to test random EO data locations on earth).
  • Ability to verify if a resource successfully finishes certain typical test cases (e.g. access data of a given size).
  • Ability to verify if a resource complies with API specification and/or profile (such as openEO API).
  • Ability to verify if a resource (specifically a published dataset) is discoverable in the Resource Catalogue (Resource Discovery BB).
  • Ability to verify if a resource (specifically a published dataset) is accessible via Data Retrieval and Visualization services (Data Access BB).
  • Ability to verify if a resource has met KPIs (pre-defined metrics to monitor a resource).
• Temporal checking execution requirements:
  • Automated checking at regular intervals (most frequent - monthly).
  • Upon issuance of request by the user.
• Alerting:
  • Receive a notification when a check does not pass (or returns unexpected value/outcome).
• User categories:
  • Developer - someone that understands how the underlying platform functionality was implemented and operates (BB developer, Application package specialist).
  • End user - someone that will want to exploit the functionality of the platform, but will not have in-depth understanding how it was implemented and operates (Platform user).

[spinto]

Hi @dovydas-an we need to generalize on top of the very specific cases you listed and extract general requirements for the BB. For example. from the listed types resources I can see these are all (except one, the hosting environment checks) generic API call/response checks, so the BB should allow the execution of programmatic tests written in Python, with custom python libraries (e.g. the OpenEO API client). From the types of checks you listed, I extract that there should be a timeout on the execution of these tests, and that they should probably return an "OK / WARNING / CRITICAL" levels, and the check developer should be able to set retries. These are the sort of requirements I would expect for the resource health BB.

[dovydas-an]

Hi @dovydas-an we need to generalize on top of the very specific cases you listed and extract general requirements for the BB. For example. from the listed types resources I can see these are all (except one, the hosting environment checks) generic API call/response checks, so the BB should allow the execution of programmatic tests written in Python, with custom python libraries (e.g. the OpenEO API client). From the types of checks you listed, I extract that there should be a timeout on the execution of these tests, and that they should probably return an "OK / WARNING / CRITICAL" levels, and the check developer should be able to set retries. These are the sort of requirements I would expect for the resource health BB.

Hi @spinto , thank you for your comment and guidance on how the requirements should be formulated. The list of expectations I included above is just there for contextual purposes only. The actual refined requirements, I will be including a bit later today.

As part of our proposal we included a list of 21 derived requirements from those that were presented in the SoW (BR125-BR131). Our intent is to continue working with them and clarify/update them according to the information we extracted above. It would be greatly appreciated, if you could review them as well.

[spinto]

Hi @dovydas-an , thank you for the answer, I have seen also now the list of the updated use cases in https://github.com/EOEPCA/resource-health/issues/12 . I will review the refined requirements once you include them. Thank you a lot!

[dovydas-an]

Low-level requirements linked to those outlined in the SoW:

• REQ-1: The Resource Health BB shall provide a means for general end-users to manage the life cycle (definition, observation, termination, ...) of checks that they own/have access to.
  • Links to SoW BR125.
• REQ-2: The Resource Health BB shall provide a means to ensure ownership/access restrictions of individual checks are respected (presumably utilising the Identity and Access Management BB).
  • Links to SoW BR125.
• REQ-3: The Resource Health BB shall ensure that the outcomes of current (active) specified checks are determined in a timely manner (meaning the execution of the checking logic is executed when necessary and that the necessary data/information is available to the check when it is executed).
  • Links to SoW BR125.
• REQ-4: The Resource Health BB shall allow specifying checks that determine the responsiveness of platform capabilities/services (such as determining ping/watchdog timer state/API timeouts/404 responses/...) sufficient for checking general availability of all/relevant other platform services/capabilities.
  • Links to SoW BR125.
• REQ-5: The Resource Health BB shall allow specifying checks that determine that resources provided by the platform meet end-user expectations (such as appearing in certain queries/enumerations, have the right metadata, have the right shape, contains expected columns, produce certain outputs ...) sufficient for all/relevant other platform services/capabilities.
  • Links to SoW BR125.
• REQ-6: The Resource Health BB shall allow specifying checks that determine the presence or absence of specified entries in resource logs (such as the absence of error 'event').
  • Links to SoW BR128.
• REQ-7: The Resource Health BB shall provide some mechanism for the timely notification of end-users, of events related to (the determination of) check outcomes (for example through a generic mechanism like web hooks, or through a capability provided by other BBs such as the notification and automation BB).
  • Links to SoW BR126.
• REQ-8: The Resource Health BB shall provide a mechanism by which end-users can specify which events to be notified of.
  • Links to SoW BR126.
• REQ-9: The Resource Health BB shall provide a means to manage the life cycle (definition, termination, ...) of checks through a REST-style API.
  • Links to SoW BR127, BR130.
• REQ-10: The Resource Health BB shall provide a means to manage the life cycle (definition, termination, ...) of checks in a textual file-based format (for use in e.g. git repositories).
  • Links to SoW BR127, BR131.
• REQ-11: The Resource Health BB shall provide a means to manage the life cycle (definition, termination, ...) of checks through a graphical user interface accessible through standard web browsers.
  • Links to SoW BR127, BR131.
• REQ-12: The Resource Health BB shall provide a means to maintain traceability to check "evidence" of an 'event' (such as location of a triggering lines in a log or timestamp of a threshold breach in a metric) when such "evidence" is available.
  • Links to SoW BR128.
• REQ-13: The Resource Health BB shall rely on capabilities of the Notification & Automation BB to ensure the timely determination of check outcomes, unless the capabilities of the Notification & Automation BB are insufficient to determine the check outcome in a timely manner (such as fast enough for ephemeral data to not have been lost).
  • Links to SoW BR129.
• REQ-14: The Resource Health BB shall rely on capabilities of the Notification & Automation BB to ensure the timely notification of check outcomes, unless the capabilities of the Notification & Automation BB are insufficient to determine the check outcome in a timely manner (such as in order to meet end-user expectations of notification latency).
  • Links to SoW BR129.
• REQ-15: The Resource Health BB shall provide an end-users graphical user interface for accessing and summarising the history of check outcomes (of checks accessible to the end user).
  • Links to SoW BR130.
• REQ-16: The Resource Health BB shall provide an API access to the history of check outcomes sufficient to reproduce the functionality exposed in the graphical user interface.
  • Links to SoW BR131.
• REQ-17: The Resource Health BB shall provide an end-users graphical user interface for accessing and summarising execution performance statistics of check logic (such as number of times executed, time since last execution, execution time, ...) sufficient for end-user to assess the sufficient "health" of a check itself (such as meeting expectations of timeliness).
  • Links to SoW BR130.
• REQ-18: The Resource Health BB shall provide an API access to execution performance statistics of check logic sufficient to reproduce the functionality exposed in the graphical user interface.
  • Links to SoW BR131.
• REQ-19: The Resource Health BB shall allow specifying timeliness requirements of individual (active) checks as requiring (re)determination/execution at a specific sequence of potentially recurrent time points.
  • Links to SoW BR129.
• REQ-20: The Resource Health BB shall allow specifying timeliness requirements of individual (active) checks as requiring (re)determination/execution on platform/component updates.
  • Links to SoW BR129.
• REQ-21: The Resource Health BB shall allow specifying timeliness requirements of individual (active) checks as requiring (re)determination/execution on resource registration changes.
  • Links to SoW BR129.
• REQ-22: The Resource Health BB shall provide a means for general end users to specify and schedule checks that confirm the availability and function of the following resource types: hosted algorithms, hosting environments, third party backends, published datasets, ML models, dashboards.
  • Links to SoW BR125.
• REQ-23: The Resource Health BB shall allow the user to issue a check upon their request.
  • Links to SoW BR129.