Sensitivity On-The-Fly - Githubissues

I have made a PR here which tries to output sensitivity visualization after each fit of the sensitivity map:

https://github.com/rhayes777/PyAutoFit/pull/1068

However, the following integration test fails:

https://github.com/Jammy2211/autolens_workspace_test/blob/main/slam/integration/source_lp/mass_total/sensitivity.py

The error is as follows:

Traceback (most recent call last):
  File "/mnt/c/Users/Jammy/Code/PyAuto/autolens_workspace_test/slam/integration/source_lp/mass_total/sensitivity.py", line 369, in <module>
    fit()
  File "/mnt/c/Users/Jammy/Code/PyAuto/autolens_workspace_test/slam/integration/source_lp/mass_total/sensitivity.py", line 355, in fit
    subhalo_result = slam.subhalo.sensitivity_imaging_lp.run(
  File "/mnt/c/Users/Jammy/Code/PyAuto/autolens_workspace_test/slam/subhalo/sensitivity_imaging_lp.py", line 665, in run
    result = sensitivity.run()
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoFit/autofit/non_linear/grid/sensitivity/__init__.py", line 183, in run
    self.visualizer_cls(sensitivity_result=sensitivity_result, paths=self.paths)
  File "/mnt/c/Users/Jammy/Code/PyAuto/autolens_workspace_test/slam/subhalo/subhalo_util.py", line 325, in __call__
    )
  File "/mnt/c/Users/Jammy/Code/PyAuto/autolens_workspace_test/slam/subhalo/subhalo_util.py", line 143, in visualize_sensitivity
    plotter.subplot_sensitivity()
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoLens/autolens/lens/sensitivity.py", line 365, in subplot_sensitivity
    log_likelihoods = self.result.figure_of_merit_array(
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoLens/autolens/lens/sensitivity.py", line 137, in figure_of_merit_array
    return self._array_2d_from(values=figures_of_merits)
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoLens/autolens/lens/sensitivity.py", line 86, in _array_2d_from
    values_reshaped = [value for values in values.native for value in values]
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoFit/autofit/non_linear/grid/grid_list.py", line 75, in native
    return np.reshape(np.array(self), self.shape)
  File "/home/jammy/venvs/PyAuto/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 285, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/home/jammy/venvs/PyAuto/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 59, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 899 into shape (30,30)

The error is caused by the visualization code I added here:

        for result in process_class.run_jobs(
            jobs, number_of_cores=self.number_of_cores
        ):
            if isinstance(result, Exception):
                raise result

            results.append(result)
            results = sorted(results)

            sensitivity_result = SensitivityResult(
                samples=[result.result.samples_summary for result in results],
                perturb_samples=[
                    result.perturb_result.samples_summary for result in results
                ],
                shape=self.shape,
                path_values=self.path_values,
            )

            if self.visualizer_cls is not None:
                self.visualizer_cls(sensitivity_result=sensitivity_result, paths=self.paths)

The problem is that SensitivityResult assumes that all results are contained, for example for a sensitivity grid of shape (30, 30), the input len of [result.result.samples_summary for result in results] must be 900.

However, because I moved SensitivityResult up from before sensitivity mapping is complete, it now receives a list with fewer than 900 entries (in this case 899, because the masking is such that only two grid cells are to be evaluated).

Can you edit the PR so that the input to samples and perturb_samples is always the right length? Everything should work from there.

rhayes777 / PyAutoFit

Sensitivity On-The-Fly #1069