search

rahulvigneswaran / Lottery-Ticket-Hypothesis-in-Pytorch

This repository contains a Pytorch implementation of the paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle and Michael Carbin that can be easily adapted to any model/dataset.
325 stars 91 forks source link

Freezing Pruned weights smaller than 1e-6 #5

Open yuezhixiong opened 4 years ago

yuezhixiong commented 4 years ago

In main.py

Freezing Pruned weights by making their gradients Zero

grad_tensor = np.where(tensor < EPS, 0, grad_tensor)

Does this also freeze the weights that have negative values? More than just weights with 0 values?

xiaosaisworking commented 4 years ago

I think it's really a bug. Once solving this problem, my experiment results can reach the original performance in the first iteration(pruning has not started). Hope to help you!

JaminLi commented 4 years ago

I think it's really a bug. Once solving this problem, my experiment results can reach the original performance in the first iteration(pruning has not started). Hope to help you!

I also meet this problem, my experiment results can't reach the original performance. How can I solve this problem with the "grad_tensor = np.where(tensor < EPS, 0, grad_tensor)" , please?

zdhNarsil commented 4 years ago

@JaminLi I think it should be grad_tensor = np.where(tensor.abs() < EPS, 0, grad_tensor).