search

zudi-lin / pytorch_connectomics

PyTorch Connectomics: segmentation toolbox for EM connectomics
http://connectomics.readthedocs.io/
MIT License
172 stars 77 forks source link

add run_chunk() to deal with tile format dataset #44

Open Michaelsqj opened 4 years ago

Michaelsqj commented 4 years ago

if the dataset is tile format, it will update and gets trained in run_chunk()

zudi-lin commented 4 years ago

Thanks for the pull request! Could you please help solve the following problems:

  1. Make the dataloader iterable. When run_chunk() is called, there is no iter(self.dataloader), which means next(self.dataloader) doesn't work at thetrain() function. You may add iter(self.dataloader) to run_chunk() or make dataloader an iterator in build_dataloader().
  2. Use self.total_num_iters in trainer. The config options (cfg) is not mutable in the scripts (see this line: https://github.com/zudi-lin/pytorch_connectomics/blob/master/scripts/main.py#L41). Therefore self.cfg.SOLVER.ITERATION_TOTAL = self.cfg.DATASET.DATA_CHUNK_ITER is not valid. Maybe add a self.total_num_iters to the Trainer? Then train() and run_chunk() need to be changed accordingly.