Illegal hardware instruction error

aliavni commented 7 years ago

I am getting [1] 4589 illegal hardware instruction kur train mnist.yml when I run kur train mnist.yml. This happens after data download. I am running this on a virtual environment (Python 3.6.0 (brew installed), Mac OS 10.12.2), and here is my pip freeze:

Jinja2==2.9.4
Keras==1.2.0
kur==0.2.0
MarkupSafe==0.23
numpy==1.12.0
pydub==0.16.7
python-magic==0.4.12
python-speech-features==0.4
PyYAML==3.12
scipy==0.18.1
six==1.10.0
Theano==0.8.2
tqdm==4.11.0

I tried installing the most recent development version with no avail. Any idea why this may be happening?

ajsyp commented 7 years ago

Not off the top of my head. Sounds like a library problem. Here are a couple diagnostic things:

What is the output of kur -vv train mnist.yml?
Do you have a .theanorc file in your home directory? If so, can you post it?
What happens if you add this line to the mnist.yml
```
settings: {backend : {optimizer: no} }
```

aliavni commented 7 years ago

@ajsyp thanks for your response.

I don't have a .theanorc in my home directory.
I am still getting the same after adding settings: {backend : {optimizer: no} } to mnist.yml.
following is kur -vv train mnist.yml output (without settings: {backend : {optimizer: no} } in mnist.yml)

[INFO 2017-01-19 14:06:13,775 kur.kurfile:632] Parsing source: mnist.yml, included by top-level.
[INFO 2017-01-19 14:06:13,781 kur.kurfile:632] Parsing source: mnist-defaults.yml, included by mnist.yml.
[INFO 2017-01-19 14:06:13,790 kur.kurfile:79] Parsing Kurfile...
[DEBUG 2017-01-19 14:06:13,790 kur.kurfile:715] Parsing Kurfile section: settings
[DEBUG 2017-01-19 14:06:13,790 kur.kurfile:715] Parsing Kurfile section: train
[DEBUG 2017-01-19 14:06:13,793 kur.kurfile:715] Parsing Kurfile section: validate
[DEBUG 2017-01-19 14:06:13,796 kur.kurfile:715] Parsing Kurfile section: test
[DEBUG 2017-01-19 14:06:13,798 kur.kurfile:715] Parsing Kurfile section: evaluate
[DEBUG 2017-01-19 14:06:13,801 kur.containers.layers.placeholder:63] Using short-hand name for placeholder: images
[DEBUG 2017-01-19 14:06:13,802 kur.containers.layers.placeholder:96] Placeholder "images" has a deferred shape.
[DEBUG 2017-01-19 14:06:13,803 kur.kurfile:715] Parsing Kurfile section: loss
[DEBUG 2017-01-19 14:06:13,832 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/train-images-idx3-ubyte.gz
[DEBUG 2017-01-19 14:06:14,214 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/train-labels-idx1-ubyte.gz
[INFO 2017-01-19 14:06:14,240 kur.providers.batch_provider:54] Batch size set to: 32
[INFO 2017-01-19 14:06:14,241 kur.providers.batch_provider:60] Maximum number of batches set to: 15
[DEBUG 2017-01-19 14:06:14,246 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/t10k-images-idx3-ubyte.gz
[DEBUG 2017-01-19 14:06:14,297 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/t10k-labels-idx1-ubyte.gz
[INFO 2017-01-19 14:06:14,301 kur.providers.batch_provider:54] Batch size set to: 32
[INFO 2017-01-19 14:06:14,301 kur.providers.batch_provider:60] Maximum number of batches set to: 100
[DEBUG 2017-01-19 14:06:14,302 kur.backend.backend:187] Using backend: keras
[INFO 2017-01-19 14:06:14,302 kur.backend.backend:80] Creating backend: keras
[INFO 2017-01-19 14:06:14,302 kur.backend.backend:83] Backend variants: none
[INFO 2017-01-19 14:06:14,302 kur.backend.keras_backend:124] No particular backend for Keras has been requested.
[DEBUG 2017-01-19 14:06:14,303 kur.backend.keras_backend:129] Only the Theano backend for Keras is installed, so we will try to use it.
[DEBUG 2017-01-19 14:06:14,303 kur.backend.keras_backend:187] Overriding environmental variables: {'KERAS_BACKEND': 'theano', 'THEANO_FLAGS': None}
[INFO 2017-01-19 14:06:14,853 kur.backend.keras_backend:192] Keras is loaded. The backend is: theano
[INFO 2017-01-19 14:06:14,854 kur.model.model:260] Enumerating the model containers.
[INFO 2017-01-19 14:06:14,854 kur.model.model:265] Assembling the model dependency graph.
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274]   Uses:
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276]   Used by: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277]   Aliases: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274]   Uses: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276]   Used by: ..activation.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277]   Aliases: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: ..activation.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274]   Uses: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276]   Used by: ..flatten.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277]   Aliases: ..activation.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274]   Uses: ..activation.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276]   Used by: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277]   Aliases: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274]   Uses: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276]   Used by: labels
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277]   Aliases: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: labels
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274]   Uses: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276]   Used by:
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277]   Aliases: labels
[INFO 2017-01-19 14:06:14,855 kur.model.model:280] Connecting the model graph.
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:311] Building node: images
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:312]   Aliases: images
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:313]   Inputs:
[DEBUG 2017-01-19 14:06:14,855 kur.containers.layers.placeholder:116] Creating placeholder for "images" with data type "float32".
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:125] Trying to infer shape for input "images"
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:143] Inferred shape for input "images": (28, 28, 1)
[DEBUG 2017-01-19 14:06:14,855 kur.containers.layers.placeholder:126] Inferred shape: (28, 28, 1)
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:382]   Value: images
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:311] Building node: ..convolution.0
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:312]   Aliases: ..convolution.0
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:313]   Inputs:
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:315]   - images: images
[1]    11608 illegal hardware instruction  kur -vv train mnist.yml

ajsyp commented 7 years ago

I'm still not certain why this would happen, and it's likely a library/compatibility issue on your system. A couple things to try:

See if the problem is in Theano by running the Theano unit tests: THEANO_FLAGS=device=cpu python -c "import theano; theano.test()".
Check if the TensorFlow backend works by adding this line to your mnist.yml: settings: {backend : {backend: tensorflow} }. You need to first install TensorFlow (preferably from within the Kur virtual environment): pip install tensorflow.

scottstephenson commented 7 years ago

@aliavni Are you still running into this same problem?

aliavni commented 7 years ago

@scottstephenson setting mnist.yml: settings: {backend : {backend: tensorflow} } resolved my issue. I get the following when I run theano unit tests with THEANO_FLAGS=device=cpu python -c "import theano; theano.test()"

kur/lib/python3.5/site-packages/theano/misc/pycuda_init.py:34: UserWarning: PyCUDA import failed in theano.misc.pycuda_init
  warnings.warn("PyCUDA import failed in theano.misc.pycuda_init")
EE.[1]    70691 illegal hardware instruction  THEANO_FLAGS=device=cpu python -c "import theano; theano.test()"

ajsyp commented 7 years ago

Looks like a Theano bug or bad Theano install. Closing the issue.

deepgram / kur

Illegal hardware instruction error #4