Overall Predictions Inspection

[faizanahemad]

Proposed Features

• Class Distribution see here

• Confusion Matrix see here

• Table of Per Class Precision-Recall-Total Counts See inspect_predictions

• Plot Acc and loss curves from Model See plot_model_history

• Show Ordered Misclassifiec classes. The classes which are most mis-classified will come on top and help us focus on those. Logic of identifying which class is most mis-classified should be based.

  mscp = {}
  for particular_class in classes:
  mis_classification_percent = Total Misclassification of particular_class/Total examples in particular_class
  mscp[particular_class] = mis_classification_percent
  # Now order by `mis_classification_percent` and show in a tabular format.

• Show Ordered Correctly classified classes. Logic similar to above.

• Show Ordered Actual Class Misclassified As Another Class Ordered. Idea: Take actual class as dog, Now dog is misclassified as cat 5%, as ship 2%, as frog 1%. Make a table of this. Input is name of actual class. Output is a table with cat-5% at the top.

• Any Other way of interpreting/visualising Confusion Matrix

• Any other way of interpreting Errors/Predictions

• Put this under https://github.com/faizanahemad/FastNet/tree/master/fastnet/inspect , make another file called model_inspection.py so you don't overlap with someone else's work

How the API looks like?

Intended Usage Once model is trained this function can be called with Model and dataset to visualize.

Limitations

Online References/Documentation was referred while making this API

• Code of Loss-Acc Curves
• Confusion Matrix and Class Distriution

Branch Name (The github branch in which you are working on this).

Note: Before raising a PR make sure that you have made an issue like this with tag: enhancement, ensure your code is tested and documented.

[faizanahemad]

Break this into Subtasks if this seems too big to do by 1 person or in 1 week.

[unography]

Facing a weird issue with scikit-learn (0.21.3)

While importing precision_recall_fscore_support from inside a file in fastnet/inspect getting an error, but it seems to work while calling the same from Python shell. Can anyone try replicating this once?

from sklearn.metrics import precision_recall_fscore_support

Stacktrace which I'm getting -

Traceback (most recent call last):
  File "/Users/dhruv-seam/Documents/FastNet/fastnet/inspect/gradcam.py", line 1, in <module>
    from sklearn.metrics import precision_recall_fscore_support, confusion_matrix, balanced_accuracy_score, accuracy_score
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/sklearn/__init__.py", line 76, in <module>
    from .base import clone
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/sklearn/base.py", line 16, in <module>
    from .utils import _IS_32BIT
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 17, in <module>
    from . import _joblib
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/sklearn/utils/_joblib.py", line 8, in <module>
    import joblib
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/joblib-0.14.0-py3.7.egg/joblib/__init__.py", line 113, in <module>
    from .memory import Memory, MemorizedResult, register_store_backend
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/joblib-0.14.0-py3.7.egg/joblib/memory.py", line 32, in <module>
    from ._store_backends import StoreBackendBase, FileSystemStoreBackend
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/joblib-0.14.0-py3.7.egg/joblib/_store_backends.py", line 16, in <module>
    from .backports import concurrency_safe_rename
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/joblib-0.14.0-py3.7.egg/joblib/backports.py", line 9, in <module>
    from pkg_resources import parse_version
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3126, in <module>
    @_call_aside
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3110, in _call_aside
    f(*args, **kwargs)
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3139, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 572, in _build_master
    ws = cls()
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 565, in __init__
    self.add_entry(entry)
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 621, in add_entry
    for dist in find_distributions(entry, True):
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1873, in find_distributions
    finder = _find_adapter(_distribution_finders, importer)
  File "/Users/dhruv-seam/Documents/FastNet/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3039, in _find_adapter
    types = _always_object(inspect.getmro(getattr(ob, '__class__', type(ob))))
AttributeError: module 'inspect' has no attribute 'getmro'