Computing attribute histograms fails when there are fewer than k nuclei

[CarlinLiao]

I've been trying to replicate the patho quant results using the BRACS data (i.e., no changes to the source code or function calls), but have been failing at multiple points. One of them is in _compute_attr_histograms in AttributeSeparability of run_qualitative_analysis.py (shouldn't this file be named run_quantitative_analysis instead?).

            attrs = [c[np.argsort(s)[-k:]] for c, s in zip(concept_list, importance_list)]
            attrs = np.concatenate(attrs, axis=0)  # (#samples x k) x #attrs 
            attrs[attrs == inf] = 0  # ensure no weird values in attributes 
            attrs = minmax_scale(attrs)   
            attrs = np.reshape(attrs, (-1, k, n_attrs))  # #samples x k x #attrs 
            attrs = list(attrs)

The error is thrown at the reshape call, which doesn't work because, when attrs is originally generated, some of the importance vectors in importance_list have less than k entries (which are equivalent to nuclei in an ROI, I think). There should be some contingency in this code block to account for if there are fewer cells than k.

That said, the real issue is probably that the nuclei detector is failing on some ROIs. I dug around a bit and noticed that the nuclei detector is finding far fewer nuclei in some ROIs that it should. For example, on BRACS_1897_DCIS_4.png it's only detecting 4 nuclei, although for this example specifically it doesn't even make it out of graph generation properly because it breaks the handcrafted nuclei concept extractor too, which is a whole 'nother issue.

Full traceback:

Traceback (most recent call last):
  File "/home/liaoc2/miniconda3/envs/hactnet_hpc/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/liaoc2/miniconda3/envs/hactnet_hpc/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/liaoc2/.vscode-server/extensions/ms-python.python-2022.16.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "run_qualitative_analysis.py", line 373, in <module>
    main(args=parse_arguments())
  File "run_qualitative_analysis.py", line 339, in main
    tumor_label_list=labels
  File "run_qualitative_analysis.py", line 97, in process
    all_histograms = self._compute_attr_histograms(nuclei_importance_list, nuclei_concept_list, tumor_label_list, n_attrs)
  File "run_qualitative_analysis.py", line 169, in _compute_attr_histograms
    attrs = np.reshape(attrs, (-1, k, n_attrs))  # #samples x k x #attrs 
  File "<__array_function__ internals>", line 6, in reshape
  File "/home/liaoc2/miniconda3/envs/hactnet_hpc/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/home/liaoc2/miniconda3/envs/hactnet_hpc/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 43848 into shape (10,24)