ERROR: an illegal memory access was encountered

[besirkurtulmus]

ubuntu@ip-xx-x-xxx-xxx:~/texture_nets$ th train.lua -data dataset -style_image style.jpg -style_size 600 -image_size 512 -model johnson -batch_size 4 -learning_rate 1e-2 -style_weight 10 -style_layers relu1_2,relu2_2,relu3_2,relu4_2 -content_layers relu4_2
torch.display not found. unable to plot
Using TV loss with weight   0
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
Successfully loaded data/pretrained/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Setting up texture layer    4   :   relu1_2
Setting up texture layer    9   :   relu2_2
Setting up texture layer    14  :   relu3_2
Setting up content layer    23  :   relu4_2
Setting up texture layer    23  :   relu4_2
        Optimize
THCudaCheck FAIL file=/tmp/luarocks-cutorch/lib/THC/generic/THCStorage.c line=182 error=77 : an illegal memory access was encountered
/home/ubuntu/torch/install/bin/luajit: cuda runtime error (77) : an illegal memory access was encountered at /tmp/luarocks-cutorch/lib/THC/generic/THCStorage.c:182

I get the following error with any given image. This was working about a week ago, I think a change in one of the libraries/packages used in torch has recently broken something.

The installation script can be found here

I've tried pegging the version of cutorch by checking out to a version 2 weeks old, but that hasn't solved the problem yet.

OS version is Ubuntu 16.04.

Any constructive criticism is appreciated.

-Besir

[DmitryUlyanov]

Hello Besir, unfortunately I have no clue, where exactly the error happens. You can try to update Torch to dev version with

luarocks install torch
luarocks install sys
luarocks install cutorch
luarocks install loadcaffe
luarocks install nn
luarocks install cunn
luarocks install optim
luarocks install image
luarocks install cudnn

after you've installed it with

  git clone https://github.com/torch/distro.git ~/torch --recursive
  cd ~/torch; bash install-deps;

Tell me if it helps.

[besirkurtulmus]

Hi Dmitry,

After reinstalling the luarocks from dev source, it seemed to work.

I've pegged the git-hash version numbers for the rocks, as it seems to be normal in the torch world for luarock maintainers to break version compatibility every now and then. (happened a couple of times in the last few months).

You can see the changes I've made here.

Thanks for all the help you've provided.

Cheers, Besir

[viekie]

I have solved the problem;

1. download protobuf 3.x compile and install, remember to load new protobuf.so file
2. remove loadcaffe and re install