Linux styleTransfer MXNet 1.2.1 Error: cudaMalloc failed: out of memory

[THEDFG]

(venv) user@user:~/Desktop/turi/styleTransfer$ pip freeze | grep mxnet mxnet==1.1.0.post0 mxnet-cu92==1.2.1 (venv) user@user:~/Desktop/turi/styleTransfer$ python styleTransfer.py WARNING: You are using MXNet 1.2.1 which may result in breaking behavior. To fix this, please install the currently recommended version:

         pip uninstall -y mxnet && pip install mxnet==1.1.0

     If you want to use a CUDA GPU, then change 'mxnet' to 'mxnet-cu90' (adjust 'cu90' depending on your CUDA version):

Read 248 images in 5.07559 secs speed: 48.8613 file/sec Read 264 images in 21.0973 secs speed: 0.998642 file/sec Using 'image' in style_dataset as feature column and using 'image' in content_dataset as feature column Premature end of JPEG file WARNING: You are using MXNet 1.2.1 which may result in breaking behavior. To fix this, please install the currently recommended version:

         pip uninstall -y mxnet && pip install mxnet==1.1.0

     If you want to use a CUDA GPU, then change 'mxnet' to 'mxnet-cu90' (adjust 'cu90' depending on your CUDA version):

Using default 1 lambda workers. To maximize the degree of parallelism, add the following code to the beginning of the program: "turicreate.config.set_runtime_config('TURI_DEFAULT_NUM_PYLAMBDA_WORKERS', 2)" Note that increasing the degree of parallelism also increases the memory footprint. /home/user/venv/local/lib/python2.7/site-packages/mxnet/gluon/block.py:413: UserWarning: load_params is deprecated. Please use load_parameters. warnings.warn("load_params is deprecated. Please use load_parameters.") Using GPU to create model (CUDA) Analyzing visual features of the style images Premature end of JPEG file Traceback (most recent call last): File "styleTransfer.py", line 24, in model = tc.style_transfer.create(styles, content, max_iterations=5000, _advanced_parameters=params) File "/home/user/venv/local/lib/python2.7/site-packages/turicreate/toolkits/style_transfer/style_transfer.py", line 339, in create cur_loss = _np.mean([L.asnumpy()[0] for L in Ls]) File "/home/user/venv/local/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 1876, in asnumpy ctypes.c_size_t(data.size))) File "/home/user/venv/local/lib/python2.7/site-packages/mxnet/base.py", line 149, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [11:00:11] src/storage/./pooled_storage_manager.h:108: cudaMalloc failed: out of memory

Stack trace returned 10 entries: [bt] (0) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f9912) [0x7f1d34e3c912] [bt] (1) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f9ee8) [0x7f1d34e3cee8] [bt] (2) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f4b6cf) [0x7f1d37a8e6cf] [bt] (3) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f4fcdc) [0x7f1d37a92cdc] [bt] (4) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x443a5b) [0x7f1d34f86a5b] [bt] (5) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x29bc8a9) [0x7f1d374ff8a9] [bt] (6) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x29bd6cc) [0x7f1d375006cc] [bt] (7) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x29bdd86) [0x7f1d37500d86] [bt] (8) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x293a313) [0x7f1d3747d313] [bt] (9) /home/user/venv/local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2942714) [0x7f1d37485714]

WARNING: You are using MXNet 1.2.1 which may result in breaking behavior. To fix this, please install the currently recommended version:

         pip uninstall -y mxnet && pip install mxnet==1.1.0

     If you want to use a CUDA GPU, then change 'mxnet' to 'mxnet-cu90' (adjust 'cu90' depending on your CUDA version):

[shreyasvj25]

Can you check how much memory your gpu has? Use nvidia-smi. If it is less, lower your batch_size to 4 or 2. params = {'batch_size': 4} model = tc.style_transfer.create(styles, content, _advanced_parameters=params)

[gustavla]

The PR #909 should address this by making it clearer to users when memory constraints are not satisfied and what to do about it.

[znation]

Dupe of #137.

[THEDFG]

with params={'input_shape': (500, 500), 'batch_size': 4} model = tc.style_transfer.create(styles, content, max_iterations=5000, _advanced_parameters=params)

(venv) user@user:~/Desktop/turi/styleTransfer$ pip freeze | grep mxnet mxnet==1.1.0 mxnet-cu92==1.2.1 (venv) user@user:~/Desktop/turi/styleTransfer$ python styleTransfer.py WARNING: You are using MXNet 1.2.1 which may result in breaking behavior. To fix this, please install the currently recommended version:

         pip uninstall -y mxnet && pip install mxnet==1.1.0

     If you want to use a CUDA GPU, then change 'mxnet' to 'mxnet-cu90' (adjust 'cu90' depending on your CUDA version):

Read 64 images in 7.28087 secs speed: 8.79016 file/sec ^NRead 272 images in 13.5198 secs speed: 33.3391 file/sec Using 'image' in style_dataset as feature column and using 'image' in content_dataset as feature column Premature end of JPEG file WARNING: You are using MXNet 1.2.1 which may result in breaking behavior. To fix this, please install the currently recommended version:

         pip uninstall -y mxnet && pip install mxnet==1.1.0

     If you want to use a CUDA GPU, then change 'mxnet' to 'mxnet-cu90' (adjust 'cu90' depending on your CUDA version):

Using default 1 lambda workers. To maximize the degree of parallelism, add the following code to the beginning of the program: "turicreate.config.set_runtime_config('TURI_DEFAULT_NUM_PYLAMBDA_WORKERS', 2)" Note that increasing the degree of parallelism also increases the memory footprint. /home/user/venv/local/lib/python2.7/site-packages/mxnet/gluon/block.py:413: UserWarning: load_params is deprecated. Please use load_parameters. warnings.warn("load_params is deprecated. Please use load_parameters.") Using 4 GPUs to create model (CUDA) Analyzing visual features of the style images [23:25:37] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

Segmentation fault: 11

Segmentation fault: 11

Segmentation fault: 11

Segmentation fault: 11

Segmentation fault (core dumped)

[znation]

Sorry @THEDFG, I misidentified the parent issue. This is actually a duplicate of #670 - the reason we segmentation fault is an incompatibility with mxnet 1.2+. Please follow #670 for updates on this, thanks!