search

rdevon / cortex

A machine learning library for PyTorch
BSD 3-Clause "New" or "Revised" License
92 stars 25 forks source link

Weight Regularization Implementation #20

Closed bbradt closed 6 years ago

bbradt commented 6 years ago

Adds two new .py files: "models/modules/regularization.py" - handles basic regularization tasks on weights using PyTorch

"models/reg_classifier.py" - a basic classifier with regularization, which defaults to L1 regularization, but can also perform L2 and EN via the command line. The regularization factor, lambda, can also be passed to the command line.

also adds the model registration to "models/init.py"

Usage: python main.py reg_classifier -S MNIST -n l1reg -a procedures.regularizer='l1',procedures.factor=0.005 python main.py reg_classifier -S MNIST -n l2reg -a procedures.regularizer='l2',procedures.factor=0.005 python main.py reg_classifier -S MNIST -n enreg -a procedures.regularizer='en',procedures.factor=0.005

Notes: L1, L2, and Elastic-Net regularization currently all work, and have been tested on MNIST data. Spectral regularization is not working for general tensors, and needs to be fixed, perhaps using the ideas discussed in Li et. al. 2016.

rdevon commented 6 years ago

Would be best to squash the commits to make the PR simpler

bbradt commented 6 years ago

@rdevon Sure, I'll squash the commits. Why don't I close this PR, clean everything up per your comments, do the squash, and make a new PR.

bbradt commented 6 years ago

@rdevon Closing pull request to squash commits