Include search request failure info in the search results notification message

[zabeen]

TBC

[zabeen]

Testing

• Env: Anthony Nolan DEV
• Commit ID: https://github.com/Anthony-Nolan/Atlas/commit/a8e581bae4aaae4fbf8d50e5c36b6d5501c4e670

Matching Algorithm - Intermittent Failure

• Simulated one type of "intermittent" failure, sql db connection failure, by setting the transient db sql connection string of the matching algorithm function app with an incorrect password, before running the search request.

First retries

Given a valid search request with valid HLA typings
And match prediction enabled,
When an intermittent failure occurs during matching algorithm request processing,
And retry attempts remain on the request,
Then multiple matching and search failure notifications should be sent out for as long as the intermittent failure lasts,
Each notification containing the correct failure information,
And saying that search will be retried.

• id: 84b7b6fd-1446-4d06-a23b-be15716c167e
• passed ✅
• Example of failure info from Matching notification:

  "FailureInfo": {
  "ValidationError": null,
  "AttemptNumber": 1,
  "RemainingRetriesCount": 9
  }

• Example of failure info from Search notification:

  "FailureInfo": {
  "StageReached": "Matching Algorithm",
  "MatchingAlgorithmFailureInfo": {
    "ValidationError": null,
    "AttemptNumber": 1,
    "RemainingRetriesCount": 9
  },
  "WillRetry": true
  }

Last Retry

Given a valid search request with valid HLA typings
And match prediction enabled,
When an intermittent failure occurs during matching algorithm request processing,
And no retry attempts remain on the request,
Then one last matching and search failure notification should be sent out,
Containing the correct failure information,
And saying that search will be NOT retried.

• id: 84b7b6fd-1446-4d06-a23b-be15716c167e
• passed ✅
• Failure info from final Matching notification:

  "FailureInfo": {
  "ValidationError": null,
  "AttemptNumber": 10,
  "RemainingRetriesCount": 0
  }

• Failure info from final Search notification:

  "FailureInfo": {
  "StageReached": "Matching Algorithm",
  "MatchingAlgorithmFailureInfo": {
    "ValidationError": null,
    "AttemptNumber": 10,
    "RemainingRetriesCount": 0
  },
  "WillRetry": false
  }

Matching Algorithm - Validation Failure

Given a valid search request with an invalid HLA typing
And match prediction enabled,
When the request is processed by the matching algorithm,
Then only a single matching and search failure notification should be sent out,
Each containing the correct failure information,
And saying that the search will NOT be retried.

• id: 1b92a8d1-4290-482f-a5ff-6af6fd6110ac
• passed ✅
• Matching notification:

  "FailureInfo": {
  "ValidationError": "Failed to lookup '*01:invalid' at locus A.",
  "AttemptNumber": 1,
  "RemainingRetriesCount": 0
  }

• Search notification:

  "FailureInfo": {
  "StageReached": "Matching Algorithm",
  "MatchingAlgorithmFailureInfo": {
    "ValidationError": "Failed to lookup '*01:invalid' at locus A.",
    "AttemptNumber": 1,
    "RemainingRetriesCount": 0
  },
  "WillRetry": false
  }

Search Orchestrator - Intermittent Failure

• Simulated one type of "intermittent" failure, sql db connection failure, by setting the match prediction sql connection string of the ATLAS-FUNCTIONS app with an incorrect password, before running the search request.

Given a valid search request with valid HLA typings
And match prediction enabled,
When an intermittent failure occurs during search orchestrator request processing,
Then the request should be retried by the orchestrator until retries are exhausted, After which only a single search failure notification should be sent out,
Containing the correct failure information,
And saying that the search will NOT be retried.

Attempt 1

• id: 1aab9c2a-3713-48e7-9bdf-2f20bafc82ef
• Test failure ❌ - search failure notification was not sent out due to NullReferenceException being thrown because Matching Algorithm Failure Info is null (as it should be).

Attempt 2

• Bug fix commit
• id: 80d68624-0b13-441c-aa2f-07570e7d01e1
• passed ✅
• Matching notification:

  "FailureInfo": null

• Search notification:

  "FailureInfo": {
  "StageReached": "RunMatchPredictionAlgorithm",
  "MatchingAlgorithmFailureInfo": null,
  "WillRetry": false
  }

Regression Testing

No Failure during Search

Given a valid search request with valid HLA typings
And match prediction enabled,
When the request has been processed,
Then a single matching and search result notification should be sent out,
Each notification containing the correct search information,
And no failure information.

• id: d5fa4423-e025-4de3-b732-e539c3d325a5
• passed ✅

Summary

• All tests but one passed on the first attempt. ✅
• Minor bug fix allowed the single test failure to pass on the second attempt. ✅