initze / thaw-slump-segmentation

MIT License
9 stars 4 forks source link

targets_only sampling mode failing #83

Closed initze closed 6 months ago

initze commented 6 months ago
data_threads: 24  # Number of threads for data loading, must be 0 on Windows
tile_size: 256 # tile size in pixels
# Sampling mode to Use. Available:
# deterministic, random, targets_only
sampling_mode: targets_only
data_sources:  # Enabled input features
  - PlanetScope
  - NDVI
  #- TCVIS
  #- RelativeElevation
  #- AbsoluteElevation
  #- Slope
  #- Hillshade
python run_train.py
WANDB_MODE=offline CUDA_VISIBLE_DEVICES='7' python train.py --data_dir /raid/scratch/initze/data_training_v6cube/planet -c configs/debug/debugv15_noAugmentRuntest.yml -wp v15_debug -wn debugv15_noAugmentRuntest -n debugv15_noAugmentRuntest

Traceback (most recent call last):
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 350, in <module>
    Engine().run()
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 85, in __init__
    for (img, mask, _) in self.get_dataloader('val'):
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 190, in get_dataloader
    self.dataset_cache[name] = get_loader(ds_config)
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/data/loading.py", line 129, in get_loader
    scenes = [NCDataset(f'{root}/{scene}.nc', config) for scene in scene_names]
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/data/loading.py", line 129, in <listcomp>
    scenes = [NCDataset(f'{root}/{scene}.nc', config) for scene in scene_names]
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/data/loading.py", line 34, in __init__
    contours = find_contours(targets)
NameError: name 'find_contours' is not defined
initze commented 6 months ago

fixed with correct import

initze commented 6 months ago

Next error:

Traceback (most recent call last):
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 350, in <module>
    Engine().run()
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 167, in run
    self.train_epoch(data_loader)
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/train.py", line 200, in train_epoch
    for iteration, (img, target, metadata) in enumerate(progress):
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/tqdm/std.py", line 1182, in __iter__
    for obj in iterable:
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/utils/data.py", line 263, in __getitem__
    base = self.dataset[idx]
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/utils/data.py", line 109, in __getitem__
    base = self.dataset[idx]
  File "/isipd/projects-noreplica/p_initze/anaconda3/envs/aicore_v09_310_v2/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 302, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/isipd/projects/p_aicore_pf/initze/code/data_loading_update/thaw-slump-segmentation/lib/data/loading.py", line 52, in __getitem__
    bbox_idx = int(torch.randint(0, len(self.bboxes), ()))
RuntimeError: random_ expects 'from' to be less than 'to', but got from=0 >= to=0
initze commented 6 months ago

fixed with https://github.com/initze/thaw-slump-segmentation/commit/8ccf34374155a2b9a0d1d477504892ccfaed294e