Troubleshoot ORCA Final Recovery Step in UAT

[krisstanton]

Complete and verify the granule Recovery process before initiating any PROD ingests. This is the last remaining item from #152

At a high level, here are the steps to be taken to test the Recovery process.

1. Create JSON that will be the input Reference: https://nasa.github.io/cumulus-orca/docs/operator/data-recovery#recovery-workflow-input-and-output-examples (See comments below for JSON that was previously tried)
2. Recovery is triggered 'manually' via AWS console Reference: https://nasa.github.io/cumulus-orca/docs/developer/deployment-guide/recovery-workflow Example State Machine Name: cumulus-kris-sbx7894-OrcaRecoveryWorkflow This State Machine should complete it's execution quickly and then there is an asynchronous process which restores the files from DEEP_ARCHIVE and moves them after the restore.
3. Wait 48 hours
4. Check the bucket where one would expect to see recovered files be present after a recovery.

Possible Starting Points:

• [ ] Double check documentation and diagram at https://nasa.github.io/cumulus-orca/docs/developer/deployment-guide/recovery-workflow for clues as to what happens AFTER the state machine has completed
• [ ] Find the name of this asynchronous process
• [ ] Try and find out if there is any logged output for this asynchronous process - to get a better idea what went wrong
• [ ] Look over the input JSON in more detail - maybe a param was missed or mis-typed somewhere. Note, much of the JSON comes directly from other places and long lists of filenames. JSON Reference: https://nasa.github.io/cumulus-orca/docs/operator/data-recovery#recovery-workflow-input-and-output-examples
• [ ] Ask in the ORCA channel if out of leads.
• [ ] With more info (such as where logs are stored, more detail on what happens with the asynchronous process), run the test again and watch for results
• [ ] Once this test works - proceed with creating a repeatable Smoke Test for this process (including command line triggering)

[krisstanton]

Details on the most recent test which did NOT result in files recovered.

-Setup the test
    Verify that the Disaster Recovery (DR Account: (csda-dr-uat-6741)) Buckets have the contents from the smoke test (This proves/proved that ORCA is doing it's storage backup properly)
        https://s3.console.aws.amazon.com/s3/buckets/csda-cumulus-cba-uat-orca-archive?region=us-west-2&prefix=planet/PSScene3Band/&showversions=false
    Go to the Sandbox bucket and change the contents of that bucket slightly
        https://s3.console.aws.amazon.com/s3/buckets/csda-cumulus-kris-sbx7894-protected-7894?prefix=planet%2FPSScene3Band%2F&region=us-west-2&showversions=false#
        -For "granuleId": "PSScene3Band-20171201_031958_0f31",
            -I deleted the entire granule (17 files total)
        -For "granuleId": "PSScene3Band-20171201_044126_100b",
            -I renamed one of the files
                Rename File
                    20171201_044126_100b_1B_Analytic_DN_metadata.xml    xml     August 2, 2023, 04:19:52 (UTC-05:00)        15.0 B      Standard
                TO
                    RENAMED__20171201_044126_100b_1B_Analytic_DN_metadata.xml   xml     August 2, 2023, 04:19:52 (UTC-05:00)        15.0 B      Standard
            -I deleted one of the files
                Delete File
                    20171201_044126_100b_1B_Analytic.tif                tif     August 2, 2023, 04:19:52 (UTC-05:00)        15.0 B      Standard

-Create JSON that will be the input
    Reference: https://nasa.github.io/cumulus-orca/docs/operator/data-recovery#recovery-workflow-input-and-output-examples
-Recovery is triggered 'manually' via AWS console (reference: https://nasa.github.io/cumulus-orca/docs/developer/deployment-guide/recovery-workflow)
    -Saw a success in the statemachine
        State Machine Name: cumulus-kris-sbx7894-OrcaRecoveryWorkflow
        https://us-west-2.console.aws.amazon.com/states/home?region=us-west-2#/statemachines/view/arn:aws:states:us-west-2:772632177894:stateMachine:cumulus-kris-sbx7894-OrcaRecoveryWorkflow
-Waited 48 hours
-Check to see if the files have been recovered.
    The expected files were not yet recovered.

[krisstanton]

When the State Machine is manually activated, there is a popup where one would paste starting JSON into it. Here is the contents of the JSON I put in on the most recent test:

ORCA/2023_10_26a__recovery_workflow.json

{
  "payload": {
    "granules": [
      {
        "granuleId": "PSScene3Band-20171201_031958_0f31",
        "version": "integrationGranuleVersion",
        "files": [
          {
            "fileName": "20171201_031958_0f31_1B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic.tif",
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894"
          },
          {
            "fileName": "20171201_031958_0f31_1B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN.tif",
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Visual.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Visual.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Visual_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Visual_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_metadata.json",
            "key": "planet/PSScene3Band/20171201_031958_0f31_metadata.json"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_cmr.json",
            "key": "planet/PSScene3Band/20171201_031958_0f31_cmr.json"
          }
        ]
      },
      {
        "granuleId": "PSScene3Band-20171201_044126_100b",
        "version": "integrationGranuleVersion",
        "files": [
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic.tif",
            "source": "storage-ss-ingest-prod-ingesteddata-uswest2/planet/PSScene3Band-20171201_044126_100b/20171201_044126_100b_3B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Visual.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Visual.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Visual_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Visual_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_metadata.json",
            "key": "planet/PSScene3Band/20171201_044126_100b_metadata.json"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_cmr.json",
            "key": "planet/PSScene3Band/20171201_044126_100b_cmr.json"
          }
        ]
      }
    ]
  },
  "meta": {
    "buckets": {
      "protected": {
        "name": "csda-cumulus-kris-sbx7894-protected-7894",
        "type": "protected"
      },
      "internal": {
        "name": "csda-cumulus-kris-sbx7894-internal-7894",
        "type": "internal"
      },
      "private": {
        "name": "csda-cumulus-kris-sbx7894-private-7894",
        "type": "private"
      },
      "public": {
        "name": "csda-cumulus-kris-sbx7894-public-7894",
        "type": "public"
      },
      "orca_default": {
        "name": "csda-cumulus-cba-uat-orca-archive",
        "type": "orca"
      }
    },
    "collection": {
      "meta": {
        "orca": {
          "defaultBucketOverride": "csda-cumulus-cba-uat-orca-archive",
          "defaultRecoveryTypeOverride": "Standard"
        },
        "s3MultipartChunksizeMb": 200
      },
      "files": [
        {
          "regex": "^(\\d{8}_\\d{6}_.+)(?:_\\dB_.+|_thumb.*|_cmr[.]json|_metadata[.]json)$",
          "sampleFileName": "20160628_204526_1_0c74_3B_Analytic_DN_metadata.xml",
          "bucket": "protected"
        }
      ]
    }
  },
  "cumulus_meta": {
    "system_bucket": "csda-cumulus-kris-sbx7894-internal-7894",
    "asyncOperationId": "1234"
  }
}

[chuckwondo]

Here's an input that worked:

{
  "payload": {
    "granules": [
      {
        "granuleId": "PSScene3Band-20171201_031958_0f31",
        "collectionId": "PSScene3Band",
        "version": "1",
        "files": [
          {
            "fileName": "20171201_031958_0f31_1B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic.tif",
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894"
          },
          {
            "fileName": "20171201_031958_0f31_1B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN.tif",
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_1B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_1B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Visual.tif",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Visual.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_3B_Visual_metadata.xml",
            "key": "planet/PSScene3Band/20171201_031958_0f31_3B_Visual_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_metadata.json",
            "key": "planet/PSScene3Band/20171201_031958_0f31_metadata.json"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_031958_0f31_cmr.json",
            "key": "planet/PSScene3Band/20171201_031958_0f31_cmr.json"
          }
        ]
      },
      {
        "granuleId": "PSScene3Band-20171201_044126_100b",
        "collectionId": "PSScene3Band",
        "version": "1",
        "files": [
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_RPC.TXT",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_RPC.TXT"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_1B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_1B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic.tif",
            "source": "storage-ss-ingest-prod-ingesteddata-uswest2/planet/PSScene3Band-20171201_044126_100b/20171201_044126_100b_3B_Analytic.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_DN_udm.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_DN_udm.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Analytic_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Analytic_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Visual.tif",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Visual.tif"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_3B_Visual_metadata.xml",
            "key": "planet/PSScene3Band/20171201_044126_100b_3B_Visual_metadata.xml"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_metadata.json",
            "key": "planet/PSScene3Band/20171201_044126_100b_metadata.json"
          },
          {
            "bucket": "csda-cumulus-kris-sbx7894-protected-7894",
            "fileName": "20171201_044126_100b_cmr.json",
            "key": "planet/PSScene3Band/20171201_044126_100b_cmr.json"
          }
        ]
      }
    ]
  },
  "config": {
    "buckets": {
      "protected": {
        "name": "csda-cumulus-kris-sbx7894-protected-7894",
        "type": "protected"
      },
      "internal": {
        "name": "csda-cumulus-kris-sbx7894-internal-7894",
        "type": "internal"
      },
      "private": {
        "name": "csda-cumulus-kris-sbx7894-private-7894",
        "type": "private"
      },
      "public": {
        "name": "csda-cumulus-kris-sbx7894-public-7894",
        "type": "public"
      },
      "orca_default": {
        "name": "csda-cumulus-cba-uat-orca-archive",
        "type": "orca"
      }
    },
    "fileBucketMaps": [
      {
        "regex": "^(\\d{8}_\\d{6}_.+)(?:_\\dB_.+|_thumb.*|_cmr[.]json|_metadata[.]json)$",
        "sampleFileName": "20160628_204526_1_0c74_3B_Analytic_DN_metadata.xml",
        "bucket": "protected"
      }
    ],
    "s3MultipartChunksizeMb": 200,
    "excludedFileExtensions": [],
    "asyncOperationId": "e3d8b981-0da4-4cad-84c5-c12d2f1094bc"
  }
}

[krisstanton]

Just observed a successful restore. The issue here was that the documentation described triggering a Recover Method that does not use the Cumulus Message Adapter (CMA) but the JSON described was for the CMA. We had to modify the JSON to the version Chuck posted above in order to make this work. We tested it again recently with the new JSON and today we are observing restored files.