data loading issue

[vict0rsch]

/home/vsch/cloudenv/lib/python3.6/site-packages/torch/nn/functional.py:1386: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
Traceback (most recent call last):
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.6.3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.6.3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/vsch/clouds/src/train.py", line 210, in <module>
    result = trainer.run_trail()81 train/d_loss:0.2497 train/L1_loss:0.3646 train/g_loss:0.3646
  File "/home/vsch/clouds/src/train.py", line 87, in run_trail
    lambda_L1=1,
  File "/home/vsch/clouds/src/train.py", line 111, in train
    for i, (coords, real_img, metos_data) in enumerate(self.trainloader):
  File "/home/vsch/cloudenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 568, in __next__
    return self._process_next_batch(batch)
  File "/home/vsch/cloudenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
IndexError: Traceback (most recent call last):
  File "/home/vsch/cloudenv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/vsch/cloudenv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/vsch/clouds/src/data.py", line 58, in __getitem__
    path = [s for s in self.paths[key] if self.ids[j] in s][0]
  File "/home/vsch/clouds/src/data.py", line 58, in <listcomp>
    path = [s for s in self.paths[key] if self.ids[j] in s][0]
IndexError: list index out of range

[vict0rsch]

+ how standard is this subsample dict?

isn't the data loader supposed to do that?

[krisrs1128]

I think this is what people do with video data loading, because they also run into memory issues. But I haven't looked at any examples.

The dataloader has no logic about reading from disk. As far as I know, it just knows how to call different indices in a dataset object, but it's the responsibility of the dataset to provide the actual samples.

I'll look into this issue in a couple of hours. I suspect I'm forming a path name incorrectly, so it's returning an empty list.