Closed aliavni closed 7 years ago
Not off the top of my head. Sounds like a library problem. Here are a couple diagnostic things:
kur -vv train mnist.yml
?.theanorc
file in your home directory? If so, can you post it?What happens if you add this line to the mnist.yml
settings: {backend : {optimizer: no} }
@ajsyp thanks for your response.
.theanorc
in my home directory. settings: {backend : {optimizer: no} }
to mnist.yml
.kur -vv train mnist.yml
output (without settings: {backend : {optimizer: no} }
in mnist.yml
) [INFO 2017-01-19 14:06:13,775 kur.kurfile:632] Parsing source: mnist.yml, included by top-level.
[INFO 2017-01-19 14:06:13,781 kur.kurfile:632] Parsing source: mnist-defaults.yml, included by mnist.yml.
[INFO 2017-01-19 14:06:13,790 kur.kurfile:79] Parsing Kurfile...
[DEBUG 2017-01-19 14:06:13,790 kur.kurfile:715] Parsing Kurfile section: settings
[DEBUG 2017-01-19 14:06:13,790 kur.kurfile:715] Parsing Kurfile section: train
[DEBUG 2017-01-19 14:06:13,793 kur.kurfile:715] Parsing Kurfile section: validate
[DEBUG 2017-01-19 14:06:13,796 kur.kurfile:715] Parsing Kurfile section: test
[DEBUG 2017-01-19 14:06:13,798 kur.kurfile:715] Parsing Kurfile section: evaluate
[DEBUG 2017-01-19 14:06:13,801 kur.containers.layers.placeholder:63] Using short-hand name for placeholder: images
[DEBUG 2017-01-19 14:06:13,802 kur.containers.layers.placeholder:96] Placeholder "images" has a deferred shape.
[DEBUG 2017-01-19 14:06:13,803 kur.kurfile:715] Parsing Kurfile section: loss
[DEBUG 2017-01-19 14:06:13,832 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/train-images-idx3-ubyte.gz
[DEBUG 2017-01-19 14:06:14,214 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/train-labels-idx1-ubyte.gz
[INFO 2017-01-19 14:06:14,240 kur.providers.batch_provider:54] Batch size set to: 32
[INFO 2017-01-19 14:06:14,241 kur.providers.batch_provider:60] Maximum number of batches set to: 15
[DEBUG 2017-01-19 14:06:14,246 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/t10k-images-idx3-ubyte.gz
[DEBUG 2017-01-19 14:06:14,297 kur.utils.package:233] File exists and passed checksum: /Users/aliavni/kur/t10k-labels-idx1-ubyte.gz
[INFO 2017-01-19 14:06:14,301 kur.providers.batch_provider:54] Batch size set to: 32
[INFO 2017-01-19 14:06:14,301 kur.providers.batch_provider:60] Maximum number of batches set to: 100
[DEBUG 2017-01-19 14:06:14,302 kur.backend.backend:187] Using backend: keras
[INFO 2017-01-19 14:06:14,302 kur.backend.backend:80] Creating backend: keras
[INFO 2017-01-19 14:06:14,302 kur.backend.backend:83] Backend variants: none
[INFO 2017-01-19 14:06:14,302 kur.backend.keras_backend:124] No particular backend for Keras has been requested.
[DEBUG 2017-01-19 14:06:14,303 kur.backend.keras_backend:129] Only the Theano backend for Keras is installed, so we will try to use it.
[DEBUG 2017-01-19 14:06:14,303 kur.backend.keras_backend:187] Overriding environmental variables: {'KERAS_BACKEND': 'theano', 'THEANO_FLAGS': None}
[INFO 2017-01-19 14:06:14,853 kur.backend.keras_backend:192] Keras is loaded. The backend is: theano
[INFO 2017-01-19 14:06:14,854 kur.model.model:260] Enumerating the model containers.
[INFO 2017-01-19 14:06:14,854 kur.model.model:265] Assembling the model dependency graph.
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274] Uses:
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276] Used by: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277] Aliases: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274] Uses: images
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276] Used by: ..activation.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277] Aliases: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:272] Assembled Node: ..activation.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:274] Uses: ..convolution.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:276] Used by: ..flatten.0
[DEBUG 2017-01-19 14:06:14,854 kur.model.model:277] Aliases: ..activation.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274] Uses: ..activation.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276] Used by: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277] Aliases: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274] Uses: ..flatten.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276] Used by: labels
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277] Aliases: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:272] Assembled Node: labels
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:274] Uses: ..dense.0
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:276] Used by:
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:277] Aliases: labels
[INFO 2017-01-19 14:06:14,855 kur.model.model:280] Connecting the model graph.
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:311] Building node: images
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:312] Aliases: images
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:313] Inputs:
[DEBUG 2017-01-19 14:06:14,855 kur.containers.layers.placeholder:116] Creating placeholder for "images" with data type "float32".
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:125] Trying to infer shape for input "images"
[DEBUG 2017-01-19 14:06:14,855 kur.model.model:143] Inferred shape for input "images": (28, 28, 1)
[DEBUG 2017-01-19 14:06:14,855 kur.containers.layers.placeholder:126] Inferred shape: (28, 28, 1)
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:382] Value: images
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:311] Building node: ..convolution.0
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:312] Aliases: ..convolution.0
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:313] Inputs:
[DEBUG 2017-01-19 14:06:14,857 kur.model.model:315] - images: images
[1] 11608 illegal hardware instruction kur -vv train mnist.yml
I'm still not certain why this would happen, and it's likely a library/compatibility issue on your system. A couple things to try:
THEANO_FLAGS=device=cpu python -c "import theano; theano.test()"
.mnist.yml
: settings: {backend : {backend: tensorflow} }
. You need to first install TensorFlow (preferably from within the Kur virtual environment): pip install tensorflow
.@aliavni Are you still running into this same problem?
@scottstephenson setting mnist.yml: settings: {backend : {backend: tensorflow} }
resolved my issue. I get the following when I run theano unit tests with THEANO_FLAGS=device=cpu python -c "import theano; theano.test()"
kur/lib/python3.5/site-packages/theano/misc/pycuda_init.py:34: UserWarning: PyCUDA import failed in theano.misc.pycuda_init
warnings.warn("PyCUDA import failed in theano.misc.pycuda_init")
EE.[1] 70691 illegal hardware instruction THEANO_FLAGS=device=cpu python -c "import theano; theano.test()"
Looks like a Theano bug or bad Theano install. Closing the issue.
I am getting
[1] 4589 illegal hardware instruction kur train mnist.yml
when I runkur train mnist.yml
. This happens after data download. I am running this on a virtual environment (Python 3.6.0 (brew installed), Mac OS 10.12.2), and here is my pip freeze:I tried installing the most recent development version with no avail. Any idea why this may be happening?