Got a TileDataset error

[AnthonyChanMarvel]

I tried to use RandomTileDataset and TileDataset in my pipeline, but TileDataset returned an error Is this a dependency problem?

[AnthonyChanMarvel]

Version of deepflash2 is 0.1.3

[matjesg]

Hi @AnthonyChanMarvel , if you're doing the Hubmap challenge you need to monkey patch the read_img method - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-train

e.g.

@patch
def read_img(self:BaseDataset, file, *args, **kwargs):
    return zarr.open(str(file), mode='r')

Does this fix the bug?

[jaideep11061982]

@matjesg

I get an unexpecetd error from learn.predict tiles. I used different loss func than CE. please help

/opt/conda/lib/python3.7/site-packages/deepflash2/learner.py in predict_tiles(self, ds_idx, dl, path, mc_dropout, n_times, use_tta, tta_merge, tta_tfms, uncertainty_estimates, energy_T)
    228                 z_eng = g_eng.empty(f.name, shape=outShape, dtype='float32')
    229                 last_file = f
--> 230             z_smx[outSlice] = smx[inSlice]
    231             z_seg[outSlice] = np.argmax(smx, axis=-1)[inSlice]
    232             if uncertainty_estimates:

/kaggle/input/zarrkaggleinstall/zarr/core.py in __setitem__(self, selection, value)
   1120 
   1121         fields, selection = pop_fields(selection)
-> 1122         self.set_basic_selection(selection, value, fields=fields)
   1123 
   1124     def set_basic_selection(self, selection, value, fields=None):

/kaggle/input/zarrkaggleinstall/zarr/core.py in set_basic_selection(self, selection, value, fields)
   1215             return self._set_basic_selection_zd(selection, value, fields=fields)
   1216         else:
-> 1217             return self._set_basic_selection_nd(selection, value, fields=fields)
   1218 
   1219     def set_orthogonal_selection(self, selection, value, fields=None):

/kaggle/input/zarrkaggleinstall/zarr/core.py in _set_basic_selection_nd(self, selection, value, fields)
   1506         indexer = BasicIndexer(selection, self)
   1507 
-> 1508         self._set_selection(indexer, value, fields=fields)
   1509 
   1510     def _set_selection(self, indexer, value, fields=None):

/kaggle/input/zarrkaggleinstall/zarr/core.py in _set_selection(self, indexer, value, fields)
   1534             if not hasattr(value, 'shape'):
   1535                 value = np.asanyarray(value)
-> 1536             check_array_shape('value', value, sel_shape)
   1537 
   1538         # iterate over chunks in range

/kaggle/input/zarrkaggleinstall/zarr/util.py in check_array_shape(param, array, shape)
    514     if array.shape != shape:
    515         raise ValueError('parameter {!r}: expected array with shape {!r}, got {!r}'
--> 516                          .format(param, shape, array.shape))
    517 
    518 

ValueError: parameter 'value': expected array with shape (924, 924, 2), got (924, 924, 1)

[i-mein]

@matjesg FYI: I faced the exact same error as Antony above, tried with @patch and versions 0.1.2, 0.1.3

Also if I define the datasets as:

train_ds = RandomTileDataset(files,   **ds_kwargs) 
valid_ds = TileDataset(files,  **ds_kwargs, is_zarr=True)

then works ok. Tracing the error has something to do with reading masks and possibly skimage (?) Any help would be much appreciated

[matjesg]

@jaideep11061982 Unfortunately, the current release of deepflash2 does not support single class predictions - it will be available in the next release. Until then, you patch the 'predict_tiles()' method (https://github.com/matjesg/deepflash2/blob/master/deepflash2/learner.py#L212) by adding the following code after line 212

if dl.c==1:
    out_act = torch.sigmoid(out)
    out_act = torch.cat([(1-out_act), out_act], dim=1)

(The code in the 'tile_shift' branch (https://github.com/matjesg/deepflash2/blob/tile_shift/deepflash2/learner.py#L244) already supports single class prediction, but also adds some more features that would probably break the current release.)

Hope this fixes your problem for now :)

[matjesg]

@i-mein Does your problem also relate to the Hubmap challenge? It is hard for me to track down the errors, but the data there needs custom preprocessing (as shown in my notebooks, via the tifffile package) before it can be loaded into the deepflash2 datasets. I hope this will fix the problem!

[i-mein]

Yes it is about Hubmap and your example notebook, I use images from hubmap-zarr (v.11) and masks from hubmap-labels-pdf-0-5-0-25-0-01 (v.4) without any other change but the error remains. I suspect something with wrong path on reading masks since when I remove label_fn works OK (but loads only images of course) - I try to found a workaround and let you know ps: I spend a lot of time reading the gist and the tutorials and It's pitty I can't run any experiments

Thanks for your help

EDIT: The exact error I got is the same ValueError as shown in the image above.

[i-mein]

Just to let you know that finally I make it work. Was either a path or a version messed up - when I started a new clean env and install latest version (0.1.3) problem fixed.

Thanks for the help