Testing and exporting commands

[bioinfo-dirty-jobs]

Dear all, I run the colab tutorial but I miss something. I have this questions: 1) How can test the model after running? 2) How can save checkpoint and the model? thanks so much

[eisen-ai]

Hello,

thanks for trying out! Eisen is very very young (< 1 month) and features are added constantly every day. Right now, some (untested) code to run validation and testing should be already in the repository (but not in the current pip package).

You can install the latest version from sources by running

pip3 install --upgrade git+https://github.com/eisen-ai/eisen-core.git

the implementation of the functionality you are looking for is here: https://github.com/eisen-ai/eisen-core/blob/master/eisen/utils/workflows/validation.py and https://github.com/eisen-ai/eisen-core/blob/master/eisen/utils/workflows/testing.py

In order to save a model, for now, you can use the standard way pytorch does it

https://pytorch.org/tutorials/beginner/saving_loading_models.html

in short torch.save(model.state_dict(), PATH)

As a note, The Colab notebook uses the MSD dataset which has only training and test set. The idea is to add now an utility function to generate splits of a given dataset so that even if MSD dataset does not have a validation set, one can just obtain it by performing a random split of the training dataset.

[bioinfo-dirty-jobs]

Thanks so much for your work and for the help!!!

When I try to update the package I have this error: ImportError: cannot import name 'GenericWorkflow' from 'eisen.utils.workflows' (/home/thor/miniconda3/envs/eisen/lib/python3.7/site-packages/eisen/utils/workflows/__init__.py)

[eisen-ai]

I think you need to re-install from the sources. In the Colab notebook you need to change the initial !pip3 install eisen-core to !pip3 install --upgrade git+https://github.com/eisen-ai/eisen-core.git execute that and restart the environment.

then you can import Validation and Testing (provided you have a dataset to validate and test) like this: from eisen.utils.workflows import Validation and from eisen.utils.workflows import Testing

[bioinfo-dirty-jobs]

I use conda enviroment and I try different python version (3.6, 3.7) If I use the pip version of github I have this error:

Python 3.6.7 | packaged by conda-forge | (default, Nov  6 2019, 16:19:42) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from eisen.utils.workflows import Validation                                                                                                                                                                                                                                
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-26058458690f> in <module>
----> 1 from eisen.utils.workflows import Validation

~/miniconda3/envs/eisen/lib/python3.6/site-packages/eisen/utils/workflows/__init__.py in <module>
----> 1 from .testing import *
      2 from .training import *
      3 
      4 
      5 class GenericWorkflow:

~/miniconda3/envs/eisen/lib/python3.6/site-packages/eisen/utils/workflows/testing.py in <module>
      8 )
      9 from eisen.utils import merge_two_dicts
---> 10 from eisen.utils.workflows import GenericWorkflow
     11 
     12 from torch import Tensor

ImportError: cannot import name 'GenericWorkflow'

In the new version search the function GenericWorkflow and in the github are not present.

[eisen-ai]

It might have been a problem of circular dependencies (rookie mistake 😬). A new commit has been just pushed.

Hopefully it will now work.

[bioinfo-dirty-jobs]

Thanks so much! I insert here the code and error I found...

test = MSDDataset(
    PATH_DATA, 
    NAME_MSD_JSON, 
    'test', 
    transform=Compose([
        read_tform, 
        resample_tform,
        to_numpy_tform,
        crop,
        map_intensities,
        threshold_labels,
        add_channel,
    ])
)

data_loader2 = DataLoader(
    test, 
    batch_size=BATCH_SIZE, 
    shuffle=True, 
    num_workers=2
)

# %%
from eisen.utils.workflows import Testing
metrics = EisenModuleWrapper(module=DiceMetric, input_names=['predictions', 'label'], output_names=['dice_metric'])

workflow2 = Testing(
      model=model,      
      data_loader=data_loader2,
      metrics=metrics,
      gpu=True

    )

hook2 = LoggingHook()
workflow2.run()

Here you have the log

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/Desktop/TEST_eisen/Test_eisen.py in 
     13 
     14 hook2 = LoggingHook()
---> 15 workflow2.run()

~/miniconda3/envs/eisen2/lib/python3.6/site-packages/eisen/utils/workflows/testing.py in run(self)
     72 
     73         with torch.no_grad():
---> 74             for i, batch in enumerate(self.data_loader):
     75                 if self.gpu:
     76                     for key in batch.keys():

~/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    817             else:
    818                 del self._task_info[idx]
--> 819                 return self._process_data(data)
    820 
    821     next = __next__  # Python 2 compatibility

~/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
    844         self._try_put_index()
    845         if isinstance(data, ExceptionWrapper):
--> 846             data.reraise()
    847         return data
    848 

~/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    383             # (https://bugs.python.org/issue2651), so we work around it.
    384             msg = KeyErrorMessage(msg)
--> 385         raise self.exc_type(msg)

TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in 
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/eisen/datasets/msd.py", line 77, in __getitem__
    sample = self.transform(item)
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/home/thor/miniconda3/envs/eisen2/lib/python3.6/site-packages/eisen/io/imaging.py", line 55, in __call__
    img = nib.load(os.path.normpath(os.path.join(self.data_dir, data[field])))
TypeError: string indices must be integers

[eisen-ai]

Thanks for trying it out.

I believe that this might be due to the absence of labels in the test set of the medical segmentation decathlon dataset.

The organisers of the medical segmentation decathlon challenge did not in fact publish the labels for the test cases.

The best thing to do would be to split the training set into training + validation. I am currently writing an utility function to accomplish this. I will try it out and let you know.

[eisen-ai]

Hello,

The notebook on Colab now includes testing: https://colab.research.google.com/drive/1aCvGYyADXPLxNqVRfFvbyN70WkMpdfDd

as a lot of the code is untested the next step for us is to focus on getting all of this tested. Thanks for putting up with the early issue of Eisen package. Feel free to contribute to the codebase as you want.