Test if pruning is possible on convex neural networks

[JacobGlennAyers]

I imagine that the output of the pruning a convex neural network is no longer convex anymore, but if we get solid results, then who really cares? It may mean that it technically generalizes better anyways. We would probably start this by plotting the weights of each layer on a histogram, and seeing what kind of distribution is showing up. Something like this:

Before: After:

[JacobGlennAyers]

Here is a result, it seems that we have what isn't quite a standard-normal distribution, as the center looks a bit too pointy, but it still looks like the majority of values are clustered around the zero mean, which can show strong results for pruning.

[JacobGlennAyers]

Probably start with fine-grain pruning and then do maybe 3-5 epochs of fine-tuning

[JacobGlennAyers]

Refer to the "global pruning" portion of this Pytorch pipeline: https://pytorch.org/tutorials/intermediate/pruning_tutorial.html