search

ddbourgin / numpy-ml

Machine learning, in numpy
https://numpy-ml.readthedocs.io/
GNU General Public License v3.0
15.35k stars 3.72k forks source link

Feature: add more activation functions #8

Closed WuZhuoran closed 5 years ago

WuZhuoran commented 5 years ago

Fix #7 , part of them.

Plot test as follow:

plot

WuZhuoran commented 5 years ago

I have some problem with softmax and hard_sigmoid function implementation.

WuZhuoran commented 5 years ago

Add documentation for each activations based on Keras documentation.

WuZhuoran commented 5 years ago

Linear units are just Affine with slope=1, intercept=0. Perhaps just make the Linear activation an instance of Affine with fixed slope and intercept?

You mean we still create Linear Class but we use Affine API to do so?

That is cool.

ddbourgin commented 5 years ago

Thanks for these! I haven't finished going through them, but if you could include unit tests against the existing PyTorch implementations (see my hacky examples here for reference), that would help a ton.

ddbourgin commented 5 years ago

Linear units are just Affine with slope=1, intercept=0. Perhaps just make the Linear activation an instance of Affine with fixed slope and intercept?

You mean we still create Linear Class but we use Affine API to do so?

That is cool.

Linear units are just Affine with slope=1, intercept=0. Perhaps just make the Linear activation an instance of Affine with fixed slope and intercept?

You mean we still create Linear Class but we use Affine API to do so?

That is cool.

Yup, you could do something like:

class Linear(Affine):
    def __init__(self):
        super().__init__(slope=1, intercept=0)

    def __str__(self):
        return "Linear"
WuZhuoran commented 5 years ago

Thanks for these! I haven't finished going through them, but if you could include unit tests against the existing PyTorch implementations (see my hacky examples here for reference), that would help a ton.

I will start new pull request for unit test.

BTW, we include all the tests in one file?

ddbourgin commented 5 years ago

BTW, we include all the tests in one file?

We... shouldn't 😬. The fact that we do is an artifact from when the neural network module was much smaller. It would be better to migrate the relevant tests to a tests.py file in the appropriate subdir and delete the tests dir entirely, though I haven't had time to do this yet.

In anticipation, I went ahead and renamed the tests.py file in the activations subdir to plots.py, since it doesn't contain any real "tests" (not really sure how it even ended up with that name in the first place, tbh!).

WuZhuoran commented 5 years ago

I already fix the hard sigmoid function but i think there is a better (beautiful) way to write the grad function.

And about PRelu, i decided to create a new pull request to talk about it. In this pull request, i removed that one.

WuZhuoran commented 5 years ago

Hard Sigmoid Numpy way of function fixed.

ddbourgin commented 5 years ago

Awesome, I think this is all set! The last thing to do is to ensure that each function passes an appropriate unit test for the forward pass + gradient step, then we can merge!

WuZhuoran commented 5 years ago

Hi @ddbourgin , I update with test!

About the test part,

  1. refact the activation test part to a new file in test subdir.
  2. add selu and leakyrelu test against with pytorch version.
  3. I found some bugs when running the test

First, when running the test, the problem occurs in __init__.py file.

E
======================================================================
ERROR: Failure: ImportError (cannot import name 'calc_pad_dims_2D' from 'utils' (/Users/zhuoran/Documents/git/pose-tracking-engine/utils.py))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/numpy/1.16.0/libexec/nose/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/usr/local/Cellar/numpy/1.16.0/libexec/nose/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/Cellar/numpy/1.16.0/libexec/nose/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/Cellar/numpy/1.16.0/libexec/nose/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py", line 244, in load_module
    return load_package(name, filename)
  File "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py", line 216, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/tests/__init__.py", line 1, in <module>
    from .tests import *
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/tests/tests.py", line 16, in <module>
    from utils import calc_pad_dims_2D, conv2D_naive, conv2D, pad2D, pad1D
ImportError: cannot import name 'calc_pad_dims_2D' from 'utils' (/Users/zhuoran/Documents/git/pose-tracking-engine/utils.py)

Then I remove this line in __init__.py, this problem solved. But when i run the softmax function, the problem is:

E
======================================================================
ERROR: tests.test_activations.test_everything
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/numpy/1.16.0/libexec/nose/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/tests/test_activations.py", line 69, in test_everything
    test_activations(N=N)
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/tests/test_activations.py", line 80, in test_activations
    test_softmax_activation(N)
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/tests/test_activations.py", line 151, in test_softmax_activation
    from layers import Softmax
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/layers/__init__.py", line 1, in <module>
    from .layers import *
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/layers/layers.py", line 5, in <module>
    from initializers import WeightInitializer, OptimizerInitializer, ActivationInitializer
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/initializers/__init__.py", line 1, in <module>
    from .initializers import *
  File "/Users/zhuoran/Documents/git/numpy-ml/neural_nets/initializers/initializers.py", line 17, in <module>
    from utils import he_normal, he_uniform, glorot_normal, glorot_uniform, truncated_normal
ImportError: cannot import name 'he_normal' from 'utils' (/Users/zhuoran/Documents/git/pose-tracking-engine/utils.py)

I think both problem can be solved by using right import method.

Then I comment softmax part, then the tests passed.

Testing started at 14:36 ...
/usr/local/bin/python3.7 /Applications/PyCharm.app/Contents/helpers/pycharm/_jb_nosetest_runner.py --target tests/test_activations.py::test_everything
Launching Nosetest with arguments /Applications/PyCharm.app/Contents/helpers/pycharm/_jb_nosetest_runner.py tests/test_activations.py:test_everything in /Users/zhuoran/Documents/git/numpy-ml/neural_nets
Testing Sigmoid activation
PASSED
...
PASSED
Testing Softmax activation
Testing Tanh activation
PASSED
...
PASSED
Testing ReLU activation
PASSED
...
PASSED
Testing LeakyRelu activation
PASSED
...
PASSED
.
----------------------------------------------------------------------
Ran 1 test in 6.248s

OK

BTW, i found that our elu test only pass when we set decimal to 6 in function assert_almost_equal, the default setting is 7, i think elu still works.

WuZhuoran commented 5 years ago

And I leave out the Linear, Exponential and Hard Sigmoid in test because

  1. Linear and Exponential do not need further evidence.
  2. pytorch does not contain Hard Sigmoid
ddbourgin commented 5 years ago

I agree with you on the ELU problem - I suspect it's a numerical precision issue. I'm not at my personal computer so I can't check right now, but I'll try to go through your tests in more detail tonight.

Thanks for wrangling the import problems - I'm sure I accidentally introduced a ton of weirdness / circular imports when I was writing individual unit tests 😕. Will take a look at the PR in more detail tonight!

yiakwy commented 5 years ago

Fix #7 , part of them.

  • [x] Linear
  • [x] Softmax
  • [x] Hard Sigmoid
  • [x] Exponential
  • [x] SELU
  • [x] SELU Test
  • [x] LeakyRelu Test

Plot test as follow:

plot

@WuZhuoran Could you look at this issue https://github.com/ddbourgin/numpy-ml/issues/17

ddbourgin commented 5 years ago

Okay, great! I've made a few updates to the documentation (e.g., remove Keras-specific references, expanded descriptions, etc.) and... Merged :-) Thanks @WuZhuoran !