search

PRBonn / bonnetal

Bonnet and then some! Deep Learning Framework for various Image Recognition Tasks. Photogrammetry and Robotics Lab, University of Bonn
MIT License
233 stars 60 forks source link

bash: /home/developer/bonnetal/deploy/devel/setup.bash: No such file or directory #4

Closed DrWillway closed 5 years ago

DrWillway commented 5 years ago

Hello everyone! I have been using bonnet for semantic segmentation before and now switching to bonnetal.

When I try to run the example it gives me the following:

developer@my-pc: /bonnetal/train/tasks/segmentation$ ./train.py -c ./config/ coco/mobilenetv2_aspp_res mobilenetv2_aspp_res.yaml mobilenetv2_aspp_res_attention.yaml developer@my-pc:/bonnetal/train/tasks/segmentation$ ./train.py -c ./config/ coco/mobilenetv2_aspp_res mobilenetv2_aspp_res.yaml mobilenetv2_aspp_res_attention.yaml developer@my-pc:/bonnetal/train/tasks/segmentation$ ./train.py -c ./config/ coco/mobilenetv2_aspp_res.yaml

INTERFACE: config yaml: ./config/coco/mobilenetv2_aspp_res.yaml log dir /home/developer/logs/2019-8-01-09:06/ model path None eval only False No batchnorm False

Commit hash (training version): b'5aed807'

Opening config file ./config/coco/mobilenetv2_aspp_res.yaml ./train.py:80: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. CFG = yaml.load(f) No pretrained directory found. Copying files to /home/developer/logs/2019-8-01-09:06/ for further reference. Images from: /cache/datasets/coco/train2017 Labels from: /cache/datasets/coco/annotations/panoptic_train2017_remap Traceback (most recent call last): File "./train.py", line 116, in trainer = Trainer(CFG, FLAGS.log, FLAGS.path, FLAGS.eval, FLAGS.no_batchnorm) File "../../tasks/segmentation/modules/trainer.py", line 68, in init workers=self.CFG["dataset"]["workers"]) File "../..//tasks/segmentation/dataset/coco/parser.py", line 377, in init drop_last=True) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 176, in init sampler = RandomSampler(dataset) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py", line 66, in init "value, but got num_samples={}".format(self.num_samples)) ValueError: num_samples should be a positive integer value, but got num_samples=0

I also got the following error while downloading docker image:

After the command: sudo nvidia-docker run -ti --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v $HOME/.Xauthority:/home/developer/.Xauthority -v /home/$USER:/home/$USER --net=host --pid=host --ipc=host tano297/bonnetal:runtime /bin/bash

I got the following: To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details. bash: /home/developer/bonnetal/deploy/devel/setup.bash: No such file or directory

Can you please help me to solve this problem? Many thanks!

tano297 commented 5 years ago

Hello,

The error from running the code is probably that the directory where you are trying to get the data is not in /cache/datasets/coco, which is probably where the config file is pointing. Can you share what you are trying to achieve? The coco data needs to be preprocessed before using it, using the generate_gt.py script.

In terms of the error with docker, it's not an error. The .bashrc script from the developer user is trying to source a catkin workspace that doesn't yet exist because you need to choose how you are going to use the deployment interface. You can dismiss this warning.

DrWillway commented 5 years ago

Thanks for the quick reply.

Yes, I specified the directory in which there are train,test and valid folders with imgs and lbls.

I am trying to run it on any dataset in order to test how it works. Do you have an example of running the train.py for semantic segmentation? As I understand the procedure is a bit different than it used to be in bonnet.

tano297 commented 5 years ago

Unfortunately, since the variety of problems is so large, even within semantic segmentation itself, the code is the best example of how to run it.

The code is structured in such a way that when you write in your config file:

dataset:
  name: "mydatasetname"

the parser is loaded from /bonnetal/train/tasks/segmentation/dataset/mydatasetname/parser.py

Therefore, if you have a new dataset, your best bet is to copy from, for example, the cityscapes one into a new directory, and modify it to read the data from your own directory structure. If you are having trouble understanding what the parser.py does, you can submit questions here and I will answer :)

tano297 commented 5 years ago

Particularly, what you need to modify is the class cityscapes(Dataset): class, to be able to pop data from your directory structure, rather than another dataset's one

DrWillway commented 5 years ago

Thanks! I will try it and let you know how it goes )

DrWillway commented 5 years ago

So I made yaml and copied python parser.py file. Created folder with rgb and labelled images but still getting error.

That is my cfg.yaml file:

training parameters

train: loss: "xentropy" # must be either xentropy or iou max_epochs: 300 max_lr: 0.01 # sgd learning rate max min_lr: 0.001 # warmup initial learning rate up_epochs: 0.01 # warmup during first XX epochs (can be float) down_epochs: 10 # warmdown during second XX epochs (can be float) max_momentum: 0.9 # sgd momentum max when lr is mim min_momentum: 0.85 # sgd momentum min when lr is max final_decay: 0.995 # learning rate decay per epoch after initial cycle (from min lr) w_decay: 0.0001 # weight decay batch_size: 4 # batch size report_batch: 1 # every x batches, report loss report_epoch: 1 # every x epochs, report validation set save_summary: False # Summary of weight histograms for tensorboard save_imgs: True # False doesn't save anything, True saves some

sample images (one per batch of the last calculated batch)

                     # in log folder

avg_N: 3 # average the N best models crop_prop: height: 480 width: 480

backbone parameters

backbone: name: "ERFNet" dropout: 0.01 bn_d: 0.1 OS: 8 # output stride train: True # train backbone? extra: extra: False

decoder: name: "ERFNet" dropout: 0.01 bn_d: 0.1 train: True # train decoder? extra: os_chan: 4: 64 2: 16 1: 16 skips: True

classification head parameters

head: name: "segmentation" dropout: 0.1

dataset (to find parser)

dataset: name: "my_dataset" location: "/home/developer/bonnetal/forest_dataset/images" workers: 3 # number of threads to get data img_means: #rgb

DrWillway commented 5 years ago

Getting this error:

sudo ./train.py -c ./config/forest/ERFNet.yaml

INTERFACE: config yaml: ./config/forest/ERFNet.yaml log dir /home/developer/logs/2019-8-01-13:24/ model path None eval only False No batchnorm False

Commit hash (training version): b'5aed807'

Opening config file ./config/forest/ERFNet.yaml No pretrained directory found. Copying files to /home/developer/logs/2019-8-01-13:24/ for further reference. Images from: /home/developer/bonnetal/forest_dataset/images/train/img Labels from: /home/developer/bonnetal/forest_dataset/images/train/lbl Inference batch size: 1 Images from: /home/developer/bonnetal/forest_dataset/images/valid/img Labels from: /home/developer/bonnetal/forest_dataset/images/valid/lbl Original OS: 8 New OS: 8 Trying to get backbone weights online from Bonnetal server. Using pretrained weights from bonnetal server for backbone Downloading: "http://www.ipb.uni-bonn.de/html/projects/bonnetal/extractors/erfnet/erfnet-87729049.pth" to /home/developer/.cache/torch/checkpoints/erfnet-87729049.pth 100.0% OS: 1 , channels: 16 OS: 2 , channels: 16 OS: 4 , channels: 64 [Decoder] os: 4 in: 128 skip: 64 out: 64 [Decoder] os: 2 in: 64 skip: 16 out: 16 [Decoder] os: 1 in: 16 skip: 3 out: 16 Using normalized weights as bias for head. No path to pretrained, using bonnetal Imagenet backbone weights and random decoder. Total number of parameters: 2251842 Total number of parameters requires_grad: 2251842 Param encoder 1913168 Param decoder 338640 Param head 34 Training in device: cuda [IOU EVAL] IGNORE: tensor([], dtype=torch.int64) [IOU EVAL] INCLUDE: tensor([0, 1]) /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [736,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [737,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [738,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [128,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [129,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [130,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [131,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [132,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [133,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [134,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [135,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [136,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [137,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [138,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [139,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [140,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [141,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [142,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [506,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [507,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [508,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [509,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [510,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [511,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [608,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [609,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [610,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [611,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [612,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [613,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [614,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [615,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [616,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [617,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [618,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [619,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [620,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [621,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [622,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [623,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [624,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [625,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [626,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [627,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [26,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [27,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [28,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [29,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [30,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [31,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [372,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [373,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [374,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [375,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [376,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [377,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [378,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [379,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [380,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [381,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [382,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [383,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [917,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [918,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [919,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [920,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [921,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [922,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [923,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [924,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [925,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [926,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [927,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [235,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [236,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [237,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [238,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [239,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [240,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [241,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [242,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [243,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [244,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [245,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [246,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [247,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [248,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [249,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [250,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [251,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [252,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [253,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [254,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [255,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [96,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [97,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [98,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [99,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [100,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [101,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [102,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [103,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [104,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [105,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [106,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [107,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [108,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [109,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [110,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [111,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [112,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [113,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [114,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [115,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [116,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [117,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [118,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [119,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [120,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [121,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [122,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [123,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [124,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [125,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [126,0,0] Assertion t >= 0 && t < n_classes failed. /bonnetal-docker-base/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , signed long , T , int, int, int, int, int, signed long) [with T = float, AccumT = float]: block: [0,0,0], thread: [127,0,0] Assertion t >= 0 && t < n_classes failed. Traceback (most recent call last): File "./train.py", line 117, in trainer.train() File "../../tasks/segmentation/modules/trainer.py", line 302, in train scheduler=self.scheduler) File "../../tasks/segmentation/modules/trainer.py", line 487, in train_epoch loss.backward() File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 107, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

tano297 commented 5 years ago

That means that your labels are not 0 or 1, so the loss is failing. You should make sure that your labels are being returned from __get_item__ being of type long and only 0 or 1.

DrWillway commented 5 years ago

I don't understand how can I check it? My labels are of two colors and images stored in png

DrWillway commented 5 years ago

Labels are like that, two coloured:

19

tano297 commented 5 years ago

If that is the case, then you need to fix your __get_item__ function so that the labels go from RGB with red in the foreground, to monochrome with 1 in the foreground. Have you understood how the parser.py gets the data? I'm sorry for the short coments, I'm on the go, from my mobile

DrWillway commented 5 years ago

It's okay, really thanks for quick reply !

I am quite new to all this things and trying to understand how it works.

tano297 commented 5 years ago

I suggest that you try to put prints of shapes and types and max and min of all your labels through the 'getitem' function. Your input needs to go from [h,w,3] shape, with red in the foreground, to [h,w] with 1 in the foreground. No other values than 0 and 1 in each pixel. Like you specified in the config file.

DrWillway commented 5 years ago

It works! Thank you very much )

tano297 commented 5 years ago

Awesome!