search

microsoft / responsible-ai-toolbox

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
https://responsibleaitoolbox.ai/
MIT License
1.33k stars 346 forks source link

RAIVisionInsights to accept torch Datasets #2308

Open microsoft-sampsa opened 1 year ago

microsoft-sampsa commented 1 year ago

Is your feature request related to a problem? Please describe.

In torch we have (standard) Dataset and Dataloader classes which work fine, however, in the multiclass image classification example in here, the only input RAI accepts, are panda tables with a filename column.

Ideally, one could use the same Datasets (which eventually uses stuff from azure datastore) as one was using to train and test the model in the first place, without the need to create yet another differently-formatted dataset scheme: a directory with files + pandas table with the 0/1 column format, etc.

Describe the solution you'd like responsibleai_vision.RAIVisionInsights accepting torch Dataset

Describe alternatives you've considered n/a

Additional context

If you follow the links in here: https://github.com/microsoft/responsible-ai-toolbox/tree/main#responsible-ai-dashboard-customization "Try the tool: model debugging of a fridge image classification model" that link is broken

A bonus question

I'd imagine you know the answer to this: as of today, azure ml does not support RAIVisionInsights? Screenshot 2023-09-01 095707

Emphasis on that small appearing toast that says "only support models with scikit-learn flavor"

EDIT: maybe it's just me, but you could really emphasize all over the place that this thing only works with "fastai" API

EDIT 2: I mean - really? How about printing the traceback when catching the Exception ..?

File /anaconda/envs/mlflow-env/lib/python3.10/site-packages/responsibleai_vision/rai_vision_insights/rai_vision_insights.py:408, in RAIVisionInsights._validate_rai_insights_input_parameters(self, model, test, target_column, task_type, classes, serializer, maximum_rows_for_test)
    406     model.predict(test_img)
    407 except Exception:
--> 408     raise UserConfigValidationException(
    409         'The model passed cannot be used for'
    410         ' getting predictions via predict()'
    411     )

UserConfigValidationException: The model passed cannot be used for getting predictions via predict()
jamesbchao commented 11 months ago

Hello, thanks for reaching out with this suggestion. I will send out a PR in the next couple days with code that allows users to use standard pytorch models and datasets with the vision dashboard. Thanks!