search

TimDettmers / sparse_learning

Sparse learning library and sparse momentum resources.
MIT License
377 stars 45 forks source link

RuntimeError: expected type torch.cuda.FloatTensor but got torch.cuda.HalfTensor #2

Closed manjunaths closed 4 years ago

manjunaths commented 5 years ago

Hello, When I run the mnist example with the command-lines below I get the following error, is fp16 not supported yet ?

# python main.py --fp16 --data=cifar10 --model=wrn-28-2
<snip>   
Traceback (most recent call last):
  File "main.py", line 235, in <module>
    main()
  File "main.py", line 211, in main
    mask.add_module(model, density=args.density)
  File "/root/workdir/SparseLearning/sparse_learning/sparselearning/core.py", line 206, in add_module
    self.init(mode=sparse_init, density=density)
  File "/root/workdir/SparseLearning/sparse_learning/sparselearning/core.py", line 115, in init
    self.apply_mask()
  File "/root/workdir/SparseLearning/sparse_learning/sparselearning/core.py", line 240, in apply_mask
    tensor.data = tensor.data*self.masks[name]
RuntimeError: expected type torch.cuda.FloatTensor but got torch.cuda.HalfTensor

Regards.

TimDettmers commented 5 years ago

I implemented fp16 support before and it worked for CIFAR-10 and MNIST, but not for the ImageNet code. I removed this functionality for now (forgot to remove this artifact). I plan to add fp16 to an upcoming release.

TimDettmers commented 4 years ago

FP16 functionality has been added in ad7811b5c451130e790124445ee16c35428b4136 with a bugfix in f8f685f1892ba8b165fef3225c3c17c91e6bbd5a.

I am currently checking if there is any degradation in performance if --fp16 is used. I should have results on this later today.

TimDettmers commented 4 years ago

I could verify that FP16 yields the same performance across models for MNIST and CIFAR-10. Thus the library has fully functional and correct FP16 support.