Saving crash diagnostics does not include file manifests

[thclark]

Feature request

Use Case

Crash diagnostics is a great feature allowing us to save inputs from questions that went wrong, and recreate their inputs locally to recreate errors quickly and easily.

Current state

Unfortunately, the process doesn't work in cases where an input_manifest is supplied, for multiple reasons:

1. The manifest that's saved in the cloud as crash diagnostics doesn't include the actual data files. You just get an empty manifest, so can't run questions that require data files.

2. The load_test_fixture_from_diagnostics function returns an instantiated manifest, which doesn't pass validation because an uninstantiated manifest is required. A workaround for this is to cast the manifest back to primitive as follows:

       (
           configuration_values,
           configuration_manifest,
           input_values,
           input_manifest,
           child_emulators,
       ) = load_test_fixture_from_diagnostics(
           path=f"{self.fixtures_path}/e1df623a-2d0e-47e5-a6e0-160516a03c50"
       )
   
       runner = Runner(
           app_src=self.app_src_path,
           twine=self.twine_path,
       )
   
       analysis = runner.run(
           input_values=input_values,
           input_manifest=input_manifest.to_primitive(),
           save_diagnostics=SAVE_DIAGNOSTICS_OFF,
       )

Proposed Solution

• [ ] Update the crash diagnostics code to correctly save files to the cloud on crash
• [x] Update documentation to reflect the latest name of the load_test_fixture_from_diagnostics function
• [ ] Update the load_test_fixture_from_diagnostics function and corresponding docs to provide a friendlier API so you literally fetch the crash diagnostics, then add a unit test with a gs::path and run it and the question will run... something like:

  # First fetch the fixture data to your machine
  octue get-diagnostics --download-data --local-path tests/data your-diagnostics-bucket/e1df623a-2d0e-47e5-a6e0-160516a03c50

  class TestClass(BaseTestCase):
  def test_e1df623a(self):
      fixture = fixture_from_diagnostics(
          "tests/data/e1df623a-2d0e-47e5-a6e0-160516a03c50"
      )
      analysis = run_fixture(
          fixture,
          app_src=self.app_src_path,
          twine=self.twine_path,
     )