Reduce Options Exposed at Minutes Level

[chadlagore]

Ticket #118–Reusing the Parameters Fixed by the Parent Model

:construction_worker: Changes

• BaseModel.save serializes the BaseModel without pickling.
• Minutes.__init__ collects a parent (BaseModel) to build from, reusing all of the "fixed" parameters that that parent necessarily enforces on the transfer model (which is the point of this PR).

:flashlight: Testing Instructions

Run the tests,

pytest --cov=minutes -vvv test

Additionally, I ran the following smoke test on the model to make sure nothing broke:

minutes = Minutes(parent='cnn')
s1, s2, s3 = Speaker('s1'), Speaker('s2'), Speaker('s3')
s1.add_audio('test/fixtures/1447')  # Audio book speakers — you need to convert to wav and put in repo if you want to train this.
s2.add_audio('test/fixtures/669')
s3.add_audio('test/fixtures/5561')
minutes.add_speakers([s1, s2, s3])
minutes.fit(verbose=2)

Train on 993 samples, validate on 490 samples
Epoch 1/15
 - 2s - loss: 1.0975 - acc: 0.3444 - val_loss: 1.0945 - val_acc: 0.6122
Epoch 2/15
 - 1s - loss: 1.0923 - acc: 0.5901 - val_loss: 1.0881 - val_acc: 0.6143
Epoch 3/15
 - 1s - loss: 1.0854 - acc: 0.7885 - val_loss: 1.0794 - val_acc: 0.8796
Epoch 4/15
 - 1s - loss: 1.0742 - acc: 0.8671 - val_loss: 1.0666 - val_acc: 0.8592
Epoch 5/15
 - 1s - loss: 1.0595 - acc: 0.8630 - val_loss: 1.0476 - val_acc: 0.8878
Epoch 6/15
 - 2s - loss: 1.0361 - acc: 0.8892 - val_loss: 1.0216 - val_acc: 0.8857
Epoch 7/15
 - 2s - loss: 1.0089 - acc: 0.8761 - val_loss: 0.9868 - val_acc: 0.9041
Epoch 8/15
 - 2s - loss: 0.9704 - acc: 0.8922 - val_loss: 0.9470 - val_acc: 0.8939
Epoch 9/15
 - 2s - loss: 0.9296 - acc: 0.8781 - val_loss: 0.8936 - val_acc: 0.9163
Epoch 10/15
 - 2s - loss: 0.8805 - acc: 0.8852 - val_loss: 0.8472 - val_acc: 0.8959
Epoch 11/15
 - 2s - loss: 0.8254 - acc: 0.8882 - val_loss: 0.7774 - val_acc: 0.9286
Epoch 12/15
 - 2s - loss: 0.7720 - acc: 0.8842 - val_loss: 0.7228 - val_acc: 0.9163
Epoch 13/15
 - 2s - loss: 0.7028 - acc: 0.9104 - val_loss: 0.6633 - val_acc: 0.9204
Epoch 14/15
 - 2s - loss: 0.6456 - acc: 0.9094 - val_loss: 0.6076 - val_acc: 0.9224
Epoch 15/15
 - 2s - loss: 0.6004 - acc: 0.9023 - val_loss: 0.5552 - val_acc: 0.9224

[Schemetrical]

Damn this is pretty cool, I'll try and review it tomorrow night 👍

[Schemetrical]

this is probably my lack of knowledge but I'm getting pytest: error: unrecognized arguments: --cov=minutes, anything I forgot to do? pipenv shell works fine.

[chadlagore]

Try removing the --cov=minutes. pipenv install should have installed pyest-cov 🤔

I'll have a look :eyes:

Edit: I had pytest install above before :joy: thats what probably messed up @Schemetrical

[Schemetrical]

When I try pytest install I get ERROR: file not found: install

I’m assuming that’s what I’m missing maybe

Le 25 mai 2018 à 07:45, Chad notifications@github.com a écrit :

Try removing the --cov=minutes. pytest install should have installed pyest-cov 🤔

I'll have a look 👀

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub, or mute the thread.

[chadlagore]

Try pipenv install Yichen!

[Schemetrical]

pipenv install -> pipenv shell -> pytestinstall (fails here)

maybe i'm missing a step

On May 25, 2018, at 09:02, Chad notifications@github.com wrote:

Try pipenv install Yichen!

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub https://github.com/ubclaunchpad/minutes/pull/122#issuecomment-392104483, or mute the thread https://github.com/notifications/unsubscribe-auth/AFHdsUXUGRQNuQs0NcxkzQIcM0QdeIJ6ks5t2CscgaJpZM4UH8u6.

[chadlagore]

On the contrary @Schemetrical, you're adding a step 😄

pipenv install   # Install dependencies
pipenv shell  # Enter environment subshell
pytest --cov=minutes -vvv test   # Run tests (no pytest install)

[Schemetrical]

I can see that pytest-cov is in the Pipfile and pipenv install and pipenv shell both are fine, somehow the test is spitting out E ModuleNotFoundError: No module named 'keras'. Additionally, --cov still does not work. I'll see if I can ask grig this weekend or maybe if you're there too. Sorry this CR is taking so long lol

Perhaps @grig-guz can approve this

[grig-guz]

@Schemetrical once I get home and try to run it myself ;)

[grig-guz]

@chadlagore I have the exact same problems as @Schemetrical , here is the log:

============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /home/grigorii/anaconda3/bin/python
cachedir: .cache
rootdir: /home/grigorii/projects/minutes, inifile:
collected 0 items / 4 errors                                                   

==================================== ERRORS ====================================
_____________________ ERROR collecting test/test_audio.py ______________________
ImportError while importing test module '/home/grigorii/projects/minutes/test/test_audio.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/lib/python3.6/site-packages/_pytest/python.py:403: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../anaconda3/lib/python3.6/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
../../anaconda3/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:213: in load_module
    py.builtin.exec_(co, mod.__dict__)
test/test_audio.py:3: in <module>
    from minutes.audio import Audio
minutes/__init__.py:1: in <module>
    from .minutes import Minutes  # noqa
minutes/minutes.py:3: in <module>
    from keras.layers import Dense
E   ModuleNotFoundError: No module named 'keras'
______________________ ERROR collecting test/test_base.py ______________________
ImportError while importing test module '/home/grigorii/projects/minutes/test/test_base.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/lib/python3.6/site-packages/_pytest/python.py:403: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../anaconda3/lib/python3.6/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
../../anaconda3/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:213: in load_module
    py.builtin.exec_(co, mod.__dict__)
test/test_base.py:6: in <module>
    from minutes.models import MINUTES_MODELS_DIRECTORY
minutes/__init__.py:1: in <module>
    from .minutes import Minutes  # noqa
minutes/minutes.py:3: in <module>
    from keras.layers import Dense
E   ModuleNotFoundError: No module named 'keras'
____________________ ERROR collecting test/test_minutes.py _____________________
ImportError while importing test module '/home/grigorii/projects/minutes/test/test_minutes.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/lib/python3.6/site-packages/_pytest/python.py:403: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../anaconda3/lib/python3.6/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
../../anaconda3/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:213: in load_module
    py.builtin.exec_(co, mod.__dict__)
test/test_minutes.py:1: in <module>
    from minutes import Minutes
minutes/__init__.py:1: in <module>
    from .minutes import Minutes  # noqa
minutes/minutes.py:3: in <module>
    from keras.layers import Dense
E   ModuleNotFoundError: No module named 'keras'
____________________ ERROR collecting test/test_speaker.py _____________________
ImportError while importing test module '/home/grigorii/projects/minutes/test/test_speaker.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/lib/python3.6/site-packages/_pytest/python.py:403: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../anaconda3/lib/python3.6/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
../../anaconda3/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:213: in load_module
    py.builtin.exec_(co, mod.__dict__)
test/test_speaker.py:1: in <module>
    import test.config as c
test/config.py:7: in <module>
    from minutes import Speaker
minutes/__init__.py:1: in <module>
    from .minutes import Minutes  # noqa
minutes/minutes.py:3: in <module>
    from keras.layers import Dense
E   ModuleNotFoundError: No module named 'keras'
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 4 error in 0.39 seconds ============================

[iKevinY]

Hmm, it's strange that Keras seems to not be being installed properly for you guys; it works fine on my machine (with a clean virtualenv and everything), and the Travis build seems to succeed. I would have suggested doing a pipenv install --dev instead, but Keras isn't even a development dependency, so pipenv install should cover it. 🤔

[chadlagore]

Indeed, Travis builds it no problem. Thanks for the log @grig-guz. Can we see a full log of the following:

$ pipenv install
$ pipenv shell
$ pipenv graph
$ which python
$ python -c "import keras; print(keras.__version__)"
$ pytest test

[grig-guz]

Sorry, keras was for some reason not in my conda environment lol. Tests pass now:

(minutes-ctwz2cxW) grigorii minutes $ pytest -vvv test
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /home/grigorii/anaconda3/bin/python
cachedir: .cache
rootdir: /home/grigorii/projects/minutes, inifile:
collected 12 items                                                             

test/test_audio.py::test_add_audio PASSED                                [  8%]
test/test_audio.py::test_samples_per_observation PASSED                  [ 16%]
test/test_audio.py::test_get_spectrograms PASSED                         [ 25%]
test/test_base.py::test_add_speakers PASSED                              [ 33%]
test/test_base.py::test_unique_speakers PASSED                           [ 41%]
test/test_base.py::test_generate_training_data PASSED                    [ 50%]
test/test_base.py::test_train PASSED                                     [ 58%]
test/test_base.py::test_save_and_load_model PASSED                       [ 66%]
test/test_base.py::test_home PASSED                                      [ 75%]
test/test_minutes.py::test_train PASSED                                  [ 83%]
test/test_speaker.py::test_speaker_add_samples PASSED                    [ 91%]
test/test_speaker.py::test_speaker_equality PASSED                       [100%]

========================== 12 passed in 4.93 seconds ===========================

However, I had to install soundfile and keras by hand, I don't think it is intended to be like that. I am not sure why keras is not being installed, but with soundfile, maybe it's the naming problem: soundfile and pysoundfile.

[chadlagore]

There are no more conda environments :) everything is managed by pipenv now.

[grig-guz]

@chadlagore still doesn't run with --cov=minutes though, if that's ok I'll approve.

[iKevinY]

Would be good to figure out the pipenv issues that @Schemetrical and @grig-guz are running into, since the promise of pipenv was to avoid these problems in the first place! 😱