The code really needs fixing

[onbigion13]

You code, especially the 3 dataset classes in dataset_generic.py, they never work when one want to get dataset splits from ids. Mayby the reason is the messy nested inheritance of the classes. How could one lets a class inherits from another class and the later class neet the instantiation of the former class?

I guess you just only run your code with specified .csv file of each split. I tried longer than half year to reproduce your code, however, you really just abandon the reproduction of your code by other researchers. If you welcome researchers to reimplement your code, how could you design you datasets classes like such a chaos?

Why must you split train, val, test splits with function among dataset class??? why not just randomly split them with pytorch or random module???

And if you want to resplit multiple whole splits, then why must you generate a iterative generator in class , instead of resplit folds upn the whole object of the class??

I can believe that I waste more than half year to study an reimplement you code and want to generate some innovation upon you jobs and want to cite you jobs in my research. While you just write your dataset class in a way that prevents anyone to reproduce or improve upon it, it's so sad and annoying.

[fedshyvana]

I agree with you that the design and implementation of the dataset class can be improved. Please note however:

• The codebase was written many years ago and meant to be a resource to accompany my publication, at a time that no similar resource was publicly available to my knowledge. Since then, it has been widely used by many studies, and in general well-received by the community. It is however not meant to be a polished product that can cater to every user's specific personal needs.

• If you have specific needs for dataloading and still want to use my codebase, you are free to implement your own dataset class (which will just be a wrapper around Pytorch's Dataset class for example), and no significant changes would be required to the rest of the codebase to support this change. For example, a minimal implementation can just look something like (assume you split your csvs to train / val / test and just use them as individual dataframes).

  class WSI_Classification_Dataset(Dataset):
  def __init__(self, 
               df, 
               data_source, 
               target_transform = None,
               index_col = 'slide_id',
               target_col = 'label', 
               use_h5 = True,
               label_map = None):
      """
      Args:
      """
      self.label_map = label_map
      self.data_source = data_source
      self.index_col = index_col
      self.target_col = target_col
      self.target_transform = target_transform
      self.data = df
      self.use_h5 = use_h5
  
  def __len__(self):
      return len(self.data)
  
  def get_ids(self, ids):
      return self.data.loc[ids, self.index_col]
  
  def get_labels(self, ids):
      return self.data.loc[ids, self.target_col]
  
  def __getitem__(self, idx):
  
      slide_id = self.get_ids(idx)
      label = self.get_labels(idx)
  
      if self.label_map is not None:
          label = self.label_map[label]
      if self.target_transform is not None:
          label = self.target_transform(label)
  
      if self.use_h5:
          feat_path = os.path.join(self.data_source, 'h5_files', slide_id + '.h5')
          with h5py.File(feat_path, 'r') as f:
              features = torch.from_numpy(f['features'][:])
              coords = torch.from_numpy(f['coords'][:])
      else:
          feat_path = os.path.join(self.data_source, 'pt_files', slide_id + '.pt')
          features = torch.load(feat_path)
          coords = []
  
      return {'features': features, 'coords': coords, 'label': label}

• You are also free to use the codebase in combination of other codebases, your custom implementations etc. in other ways too. Such as just using CLAM for data processing, and use your own code for models / training or vice versa.

[onbigion13]

Thanks for your intime reply. I understand that your implemented is done years ago, and appreciate your efforts of letting your code be open to community. Your code about preprocessing the .svs slides really help a lot.

While the class about datasets for pack extracted features into Dataset really did not work when one chooses to make folds by ids, instead of .cvs files, which you adopted in your implementation. The main trouble is from the nested inheritance of multiple classes, especially when one wants to create datasets splits from the Generic_MIL_Dataset. It shows error that the arguments assigned to child class don't exist in parent class, maybe the reason lies in that you nested the inheritance of classed and did not declare the super method in Generic_Split. Even I added the super method in the constructor method of Generic_Split, more errors about member properties arised. So I believe that maybe it's your decision after consideration, and so you did not declare the super method.

And I have implemented the dataset with common ways with my own codes, and conduct the multiple folds' creation outside the class, with Random and Sub-Dataset class. Any way, thanks for your efforts for creating and sharing the repos.