lesion-predict - input not on gpu

lesion-predict --config predict_config_noAC.yaml 
Global seed set to 0
2021-11-16 19:56:36,770 - py.warnings - WARNING - /home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/data.py:1071: UserWarning: pet not in known ('ct', 'flair', 'pd', 't1', 't1c', 't2', 'label', 'weight', 'div', 'out'). Assuming pet is a non-label image.
  f"{name} not in known {RECOGNIZED_NAMES}. "

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
2021-11-16 19:56:39,967 - py.warnings - WARNING - /home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:111: UserWarning: The dataloader, predict_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."

Predicting: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/bin/lesion-predict", line 33, in <module>
    sys.exit(load_entry_point('tiramisu-brulee', 'console_scripts', 'lesion-predict')())
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/cli/predict.py", line 173, in predict
    _predict(args, parser, True)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/cli/predict.py", line 263, in _predict
    _predict_patch_image(args, model_path, model_num, p3d)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/cli/predict.py", line 236, in _predict_patch_image
    trainer.predict(model, datamodule=dm)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in predict
    self._predict_impl, model, dataloaders, datamodule, return_predictions, ckpt_path
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _predict_impl
    results = self._run(model, ckpt_path=self.predicted_ckpt_path)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
    self._dispatch()
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1270, in _dispatch
    self.training_type_plugin.start_predicting(self)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 210, in start_predicting
    self._results = trainer.run_stage()
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1281, in run_stage
    return self._run_predict()
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_predict
    return self.predict_loop.run()
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/loops/dataloader/prediction_loop.py", line 92, in advance
    dataloader_iter, self.current_dataloader_idx, dl_max_batches, self.num_dataloaders, self.return_predictions
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py", line 100, in advance
    self._predict_step(batch, batch_idx, dataloader_idx)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py", line 133, in _predict_step
    predictions = self.trainer.accelerator.predict_step(step_kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 252, in predict_step
    return self.training_type_plugin.predict_step(*step_kwargs.values())
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 225, in predict_step
    return self.model.predict_step(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/seg.py", line 268, in predict_step
    return self._predict_patch_image(batch)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/seg.py", line 346, in _predict_patch_image
    pred = self(src)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/experiment/seg.py", line 140, in forward
    out: Tensor = self.network(tensor)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tiramisu_brulee/model/tiramisu.py", line 167, in forward
    out = self.first_conv(x)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
Exception ignored in: <function tqdm.__del__ at 0x7fd4ea1df0e0>
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tqdm/std.py", line 1134, in __del__
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tqdm/std.py", line 1281, in close
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tqdm/std.py", line 1467, in display
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tqdm/std.py", line 1137, in __repr__
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/tqdm/std.py", line 1424, in format_dict
TypeError: cannot unpack non-iterable NoneType object
jcreinhold / tiramisu-brulee

lesion-predict - input not on gpu #19
