search

rdevon / cortex

A machine learning library for PyTorch
BSD 3-Clause "New" or "Revised" License
92 stars 25 forks source link

Regularization Module Squashed Commit #22

Closed bbradt closed 6 years ago

bbradt commented 6 years ago

@rdevon New PR, with same changes, squashed commit. I omitted the commits for the CAE, since it would be better to handle the PRs one feature at a time.

Message from previous PR: " Adds two new .py files: "models/modules/regularization.py" - handles basic regularization tasks on weights using PyTorch

"models/reg_classifier.py" - a basic classifier with regularization, which defaults to L1 regularization, but can also perform L2 and EN via the command line. The regularization factor, lambda, can also be passed to the command line.

also adds the model registration to "models/init.py"

Usage: python main.py reg_classifier -S MNIST -n l1reg -a procedures.regularizer='l1',procedures.factor=0.005 python main.py reg_classifier -S MNIST -n l2reg -a procedures.regularizer='l2',procedures.factor=0.005 python main.py reg_classifier -S MNIST -n enreg -a procedures.regularizer='en',procedures.factor=0.005

Notes: L1, L2, and Elastic-Net regularization currently all work, and have been tested on MNIST data. Spectral regularization is not working for general tensors, and needs to be fixed, perhaps using the ideas discussed in Li et. al. 2016. "

rdevon commented 6 years ago

So also check out changes made in https://github.com/rdevon/cortex2.0/pull/32 There are some useful functions in https://github.com/rdevon/cortex2.0/pull/32/files#diff-9ef4ccbc0bf641864ee02c639b7d406f now that are useful for making autoencoders.