Closed dorimedini closed 5 years ago
@dorimedini are you sure that you are returning the percentage? please not that the layers are different in size: the first layer is biggest (FC from 32*32 to h_dim), two middle ones are h_dim to h_dim, and the last one is smallest (h_dim to 10). Maybe you are not calculating the total number of edges properly?
@dorimedini are you sure that you are returning the percentage? please not that the layers are different in size: the first layer is biggest (FC from 32*32 to h_dim), two middle ones are h_dim to h_dim, and the last one is smallest (h_dim to 10). Maybe you are not calculating the total number of edges properly?
This explains why the results make sense... I returned the negation of the correct value in total_edges_in_layers
and then computed the negation of what I meant when I wrote pruned_percents.append(total_nonzero / total)
:D fixing...
@dorimedini are you sure that you are returning the percentage? please not that the layers are different in size: the first layer is biggest (FC from 32*32 to h_dim), two middle ones are h_dim to h_dim, and the last one is smallest (h_dim to 10). Maybe you are not calculating the total number of edges properly?
Fixed
@dorimedini how are the new graphs looking?
@dorimedini how are the new graphs looking?
@galshachaf with 30% prune threshold, these are the results. I'll open a task to show the accuracy of the trained+unpruned model vs. the trained+pruned model to make sure I'm not pruning too much / not enough
Results! ...are weird, I'm assuming initial weight sampling is not uniform in some interval? Different layers are given different-magnitude random initial weights? Otherwise we should see a fixed percentage of each layer (the threshold percent) @noamloya @galshachaf thoughts?
Addresses #97