Evaluation suite - loading input

[csudre]

Context Evaluation suite - Need to be able to address multiple input situations - Either a single pair of reference / output or a folder of matching pairs or numpy arrays already loaded to memory.

Describe the solution you'd like Flexible evaluation suite where input can be a specified pair of images to load / a folder with subject matching pairs Should also be able to handle existing numpy arrays stored in memory

https://github.com/Project-MONAI/MONAI/wiki/Evaluation-metrics-task-force

[wyli]

thanks @csudre, for those three use cases, the basic evaluation logic would be:

1. user-specified pair of images to load:

   datalist = [{'img': 'subject-1.nii.gz', 'seg': 'subject-1-seg.nii.gz'}]
   dataset = monai.data.Dataset(datalist, transform=preprocessing)
   loader = DataLoader(dataset, shuffle=False)
   results = [compute_xx_metric(item) for item in loader]

2. user-specified folder:

   data_folder = 'myfolder/'
   datalist = folder_to_list(data_folder)
   dataset = monai.data.Dataset(datalist, transform=preprocessing)
   loader = DataLoader(dataset, shuffle=False)
   results = [compute_xx_metric(item) for item in loader]

3. in memory arrays:

   data_arrays = [{'img': some_nparray, 'seg': some_nparray}]
   dataset = monai.data.Dataset(data_arrays, transform=preprocessing)
   loader = DataLoader(dataset, shuffle=False)
   results = [compute_xx_metric(item) for item in loader]

looks like what's missing just now is

• [ ] a folder_to_list utility to extract a data_list such as datalist = [{'img': 'subject-1.nii.gz', 'seg': 'subject-1-seg.nii.gz'}] from a user provided folder by listdir and match the filenames.
• [ ] a utility function to write results into a user-friendly format (csv?) or print results properly to the CLI what do you think @Nic-Ma @csudre